Methods, computer-accessible medium and systems for facilitating dark flash photography

ABSTRACT

Exemplary embodiments of the present disclosure relate generally to methods, computer-accessible medium and systems for dark flash photography. For example, described herein is an exemplary embodiment of an apparatus for providing illumination and obtaining an image, which can include, e.g., a first arrangement configured to emit a flash of light including ultra-violet light and/or infra-red light to illuminate a scene and/or one or more subjects, and a second arrangement configured to obtain an image of the illuminated scene and/or one or more subject. A duration of time and/or an intensity level of the flash can be selected, e.g., so that wavelengths of the flash can be substantially invisible to a human eye. For example, the visibility of the flash perceived by the human eye can be, e.g., approximately 200 times less than the visibility of a standard flash of light having substantially the same amount of energy as the flash.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application relates to and claims priority from U.S.Non-Provisional application Ser. No. 13/133,822 filed on Aug. 22, 2011,International Patent Application No. PCT/US2010/020511 filed on Jan. 8,2010, and from U.S. Patent Application Ser. No. 61/134,684 filed on Jan.9, 2009 and 61/241,300 filed on Sep. 10, 2009, the entire disclosures ofwhich are hereby incorporated herein by references.

FIELD OF THE DISCLOSURE

Exemplary embodiments of the present disclosure relate generally tomethods, computer-accessible medium and systems for facilitating darkflash photography.

BACKGROUND INFORMATION

Camera flashes can produce intrusive bursts of light that disturb ordazzle. Provided herein is an exemplary camera and flash that can useinfra-red and ultra-violet light outside the visible range to captureand/or obtain pictures and/or images in relatively low-light conditions.This “dark” flash can be, e.g., at least two orders of magnitude dimmerthan conventional flashes for a comparable exposure. Building on ideasfrom flash/no-flash photography, a pair of images can be captured and/orobtained, one using dark flash, the other using dim ambient illuminationalone. The relationships and/or correlations between images recorded atdifferent wavelengths can be used to denoise the ambient image andrestore fine details to give a high quality result, even in very weakillumination. The processing techniques can also be used to denoiseimages captured with conventional cameras.

The heavy-tailed distribution of gradients in natural scenes can haveproven effective priors for certain problems such as denoising,deblurring and super-resolution. These distributions can be well modeledby a hyper-Laplacian (p(x)∝e^(−k|x|) ^(α) ) typically with 0.5≦α≦0.8.However, the use of sparse distributions can make the problem non-convexand impractically slow to solve for multi-megapixel images.

The introduction of digital camera sensors has transformed photography,permitting new levels of control and flexibility over the imagingprocess. Coupled with less expensive computation power, variousphotographic techniques have been described, collectively known asComputational Photography. Modern camera sensors, whether in a cellphone or a high-end DSLR, typically use either a CCD or CMOS sensorbased on silicon. The raw sensor material can respond to light over awide range of wavelengths, which can typically be, e.g., approximately350-1200 nanometers (nm). Colored dyes can be deposited onto the sensorpixels in a Bayer pattern, resulting in 3 groups of pixels (e.g., red,green and blue). Each group responds to a limited range of wavelengths,approximating the sensitivities of the three types of cone cells in thehuman retina, for example. However, silicon is highly sensitive toinfra-red (IR) wavelengths and it therefor can be difficult tomanufacture dyes that have sufficient attenuation in this region, thusan extra filter is typically placed on top of most sensors to, e.g.,block IR light. This yields a sensor that can record only over the rangeof approximately 400-700 nm. While matching the typical human's colorperception, it is generally a considerable restriction of the intrinsicrange of the device.

One solution to capturing photographs in low light conditions is to usea flash unit to add light to the scene. Although such solution providesthe light to capture otherwise unrecordable scenes, the flash makes thephotographic process intrusive. The sudden burst of light not onlyalters the illumination but typically disturbs people present, makingthem aware that a photo has just been taken and possibly dazzling themif they happen to be looking toward the camera. For example, a groupphoto in a dark restaurant using a bright camera flash can leave thesubjects unable to see clearly for some moments afterward.

Dark flash camera/flash systems can be based around off-the-shelfconsumer equipment, with a number of minor modifications. First, thecamera can be a standard DSLR with the IR-block filter removed, thusrestoring much of the original spectral range of the sensor. Second, amodified flash can be used that emits light over a wider spectral rangethan normal, which can be filtered to remove visible wavelengths. Thisdark flash can allow for the addition of light to the scene in such away that it can be recorded by the camera, but not by a human's visualsystem. Using the dark flash, it is possible to illuminate a dimly litscene without dazzling people present or disturbing other people inclose proximity. Furthermore, it can allow for a fast shutter speed tobe used, thus avoiding camera shake. People typically want images withcolors that substantially match their own visual experience. However,this is generally not the case for images captured using heretoforeavailable flash technologies.

Exemplary embodiments in accordance with the present disclosure can beregarded as a multi-spectral variation of the flash/no-flash techniqueintroduced by Agrawal et al., Removing photography artifacts usinggradient projection and flash-exposure sampling, ACM Transactions onGraphics (Proc. SIGGRAPH), 24, 828-835 (2005), Petschnigg et al.,Digital photography with flash and no-flash image pairs, ACMTransactions on Graphics (Proc. SIGGRAPH) 23, 3, 664-672 (2004), andEisemann et al., Flash photography enhancement via intrinsic relighting,ACM Transactions on Graphics (Proc. SIGGRAPH) 23, 673-678 (2004).Agrawal et al. 2005 focused on the removal of flash artifacts but didnot apply their method to ambient images containing significant noise,unlike those described in Petschnigg et al., supra, and Eisemann et al.,supra. The approaches described in the two latter publications aresimilar to one another in that they use a cross-bilateral (also known asjoint-bilateral) filter and detail transfer. However, Petschnigg et al.,supra, attempts to denoise the ambient image, adding detail from theflash, while Eisemann et al., supra, alter the flash image using ambienttones.

Bennett et al., Multispectral bilateral video fusion, IEEE Trans. ImageProcessing 16, 5, 1185-1194 (2007), describes how video captured inlow-light conditions can be denoised using continuous IR illumination.However, they make use of temporal smoothing to achieve high qualityresults, something that is generally not possible in a photographysetting. Wang, O., et al. Video relighting using infrared illumination,Computer Graphics Forum 27 (2008), describes, e.g., how IR illuminationcan be used to relight faces in well-lit scenes. Both of theseapproaches significantly differ from exemplary embodiments in accordancewith the present disclosure in a number of ways: (i) they use complexoptical bench based setups with twin cameras and beam-splitters asopposed to a single portable DSLR camera and temporally multiplex; (ii)both use IR alone rather than the near-UV and IR (for achieving highquality reconstructions); (iii) both rely on cross-bilateral filteringto combine the IR and visible signals, an approach which can havesignificant shortcomings. In contrast, disclosed herein is a principledmechanism for propagating information between spectral bands. This canbe integrated into a unified cost function that combines the denoisingand detail transfer mechanisms, treated separately in cross-bilateralfiltering and related methods, such as is described in Farbman et al.,Edge-preserving decompositions for multi-scale tone and detailmanipulation, ACM Transactions on Graphics (Proc. SIGGRAPH) 27, 671-680(2008).

Infra-red imaging has a history in areas such as astronomy andnight-vision. In consumer photography the most prominent use can beconsidered to have been the Sony Nightshot where the IR-block filter canbe switched out to use the near-IR part of the spectrum. The images aremonochrome (with a greenish tint) and generally no attempt is made torestore natural colors to them. Other imaging approaches use Far-IRwavelengths to record the thermal signature of people or vehicles.However, this can require specialized optics and sensors and thus haslimited relevance to consumer photography. Ultra-violet (UV) photographygenerally has received little attention, other than from flowerphotography enthusiasts (see, e.g., Rorslett, B., Flowers inUltraviolet, available at http://www.naturfotograf.com/UV flowerslist.html (last accessed Jan. 7, 2010)). Many flowers that can lookplain to humans can have vibrant patterns under UV light to attractinsects sensitive to these wavelengths.

Multi-spectral recording using visible wavelengths has been explored byseveral authors. Park et al., Multispectral Imaging Using MultiplexedIllumination, ICCV 1-8 (2007), describes the use multiplexedillumination via arrays of colored LEDs to recover spectral reflectancefunctions of the scene at video frame rates. Exemplary embodiments ofthe system, method and computer-accessible medium according to thepresent disclosure can be used in a similar manner for still scenes,being able to estimate the reflectance functions beyond the visiblerange. Mohan et al., Agile spectrum imaging: Programmable wavelengthmodulation for cameras and projectors, Computer Graphics Forum 27, 2,709-717 (2008), describes use of a diffraction grating in conjunctionwith an LCD mask to give control over the color spectrum forapplications including metamer detection and adaptive color primaries.

Processing of the flash/no-flash pair in accordance with the presentdisclosure exploits the relationships and/or correlations between nearbyspectral bands. Most work on image priors can be considered as havingfocused on capturing spatial correlations within a band. For example,priors based on the heavy tailed distributions of image gradients haveproven effective in a wide range of problems such as denoising (see,e.g., Portilla et al., Image denoising using a scale mixture ofGaussians in the wavelet domain, IEEE Trans. Image Processing 12, 11,1338-1351 (2003), deblurring (see, e.g., Fergus et al., Removing camerashake from a single photograph, ACM Transactions on Graphics (Proc.SIGGRAPH) 25, 787-794 (2006)), separating reflections (see, e.g., Levinand Weiss, User assisted separation of reflections from a single imageusing a sparsity prior, IEEE Trans. Pattern Analysis and MachineIntelligence 29, 9, 1647-1654 (2007)). However, models that exploitdependencies between color channels tend to be less common. The K-SVDdenoising approach of Aharon et al., The KSVD: An algorithm fordesigning of overcomplete dictionaries for sparse representation, IEEETrans. Signal Processing 54, 11, 4311-4322 (2006), can do so by vectorquantizing color patches. The fields-of-experts approach of Roth et al.,Fields of Experts: A Framework for Learning Image Priors, CVPR, 2,860-867 (2005) has also been extended to model color images (see, e.g.,McAuley et al., Learning high-order MRF priors of color images, ICML 06,617-624 (2006)) and uses color marginal filters. However, neither ofthese approaches explicitly model the interchannel correlations, unlikethe exemplary system, method and computer accessible medium according tothe present disclosure. Explicit spectral models are used in colorconstancy problems and joint spatial-spectral models have been proposed(see, e.g., Singh et al., Exploiting spatial and spectral imageregularities for color constancy, Workshop on Statistical andComputational Theories of Vision (2003), and Chakrabarti et al., Colorconstancy beyond bags of pixels, CVPR, 1-6 (2008)) for this task, butthese generally assume a noise-free image. Morris et al., Statistics ofinfrared images, CVPR, 1-7 (2007), describes measuring the spatialgradients of far IR images gathered with a specialized camera,demonstrating their similarity to those of visible light images.

Flash-based methods are generally not the only solution to takingpictures in low-light levels. Wide aperture lenses gather more light butare heavy and expensive, making them impractical for most photographers.Anti-shake hardware can be used to capture blur-free images at slowshutter speeds. These techniques can be combined with an exemplaryapproach in accordance with the present disclosure to extend performanceto even lower light levels. Software-based deblurring techniques (see,e.g., Fergus et al., supra, and Jiaya, J., Single image motiondeblurring using transparency, CVPR, 1-8 (2007)) can only cope withmodest levels of blur and typically have artifacts in their output.Denoising techniques, such as described in, e.g., Tomasi et al.,Bilateral filtering for gray and color images, ICCV, 839-846 (1998), andPortilla et al., supra, can have similar performance issues and cannotcope with the noise levels that can be addressed by certain exemplaryembodiments disclosed herein. Joint denoising/deblurring techniques,such as that described in, e.g., Yuan et al., Image deblurring withblurred/noisy image pairs, ACM Transactions on Graphics (Proc. SIGGRAPH)26, 1-10 (2007), may provide better performance but still require aproblematic deconvolution operation, which can introduce artifacts.Methods that register and combine a stack of noisy images, such asdescribed in, e.g., Telleen et al., Synthetic shutter speed imaging,Computer Graphics Forum 26, 3, 591-598 (2007), can have theinconvenience of needing to capture far more than two images. A visibleflash can be made non-dazzling by using a diffuser and aiming at theceiling. Such methods can work appropriately but can be limited toindoor settings with a relatively low ceiling of neutral color.

Natural image statistics are a powerful tool in image processing,computer vision and computational photography. Denoising (see, e.g.,Portilla et al., supra), deblurring (see, e.g., Fergus et al., supra),transparency separation (see, e.g., Levin and Weiss, supra) andsuper-resolution (see, e.g., Tappen, M. F. et al., Exploiting the sparsederivative prior for super-resolution and image demosaicing, SCTV(2003)), are all tasks that can be inherently ill-posed. Priors based onnatural image statistics can regularize these problems to yield qualityresults. However, digital cameras now have sensors that record imageswith tens of megapixels (MP), e.g., the latest Canon DSLRs have over 20MP. Solving the above tasks for such images in a reasonable time frame(e.g., a few minutes or less), poses a significant challenge to existingalgorithms. An exemplary problem can be addressed by the exemplaryembodiments of the present disclosure, e.g., non-blind deconvolution,and can address very large images while still yielding high qualityresults.

Various deconvolution approaches can exist, varying substantially intheir speed and sophistication. Simple filtering operations are fast buttypically yield poor results. Most of the adequately-performingapproaches solve globally for the corrected image, encouraging themarginal statistics of a set of filter outputs to match those ofuncorrupted images, which can act as a prior to regularize the problem.For these methods, a trade-off can exist between accurately modeling theimage statistics and being able to solve the ensuing optimizationproblem efficiently. If the marginal distributions can be assumed to beGaussian, a closed-form solution exists in the frequency domain and FFTscan be used to recover the image very quickly. However, real-worldimages can typically have marginals that are non-Gaussian, and thus theoutput can often be of mediocre quality. One approach is to assume themarginals have a Laplacian distribution. This can allow a number of fastl₁ and related TV-norm methods, such as described in, e.g., L. Rudin etal., Nonlinear total variation based noise removal algorithms, Physica D60, 259-268 (1992), and Wang, Y. et al., A new alternating minimizationalgorithm for total variation image reconstruction, SIAM J. ImagingSciences 1, 3, 248-272 (2008), to be deployed, which can giveappropriate results in a reasonable time.

However, studies of real-world images have shown the marginaldistributions have significantly heavier tails than a Laplacian, beingmodeled by a hyper-Laplacian (see, e.g., Field, D., What is the goal ofsensory coding?, Neural Computation 6, 559-601 (1994), Levin, Fergus,Durand and Freeman, Image and depth from a conventional camera with acoded aperture, ACM TOG (Proc. SIGGRAPH) 26, 3, 70 (2007) and Simoncelliet al., Noise removal via Bayesian wavelet coring, ICIP 379-382 (1996)).Although such priors can give appropriate quality results, they cantypically be slower than methods that use either Gaussian or Laplacianpriors. This can be a consequence of the problem becoming non-convex forhyper-Laplacians with α<1, meaning that it is possible that many of thefast l₁ or l₂ tricks are no longer applicable. Instead, standardoptimization methods such as conjugate gradient (CG) can be used. Onevariant that can work in practice is iteratively reweighted leastsquares (IRLS) as described in, e.g., Stewart, C. V., Robust parameterestimation in computer vision, SIAM Reviews 41, 3, 513-537 (1999), whichcan solve a series of weighted least squares problems with CG, each onean l₂ approximation to the non-convex problem at the current point. Inboth cases, typically hundreds of CG iterations can be used, each ofwhich can involve an expensive convolution of the blur kernel with thecurrent image estimate.

Hyper-Laplacian image priors have been used in a range of settings:super-resolution (see, e.g., Tappen et al., supra), transparencyseparation (see, e.g., Levin and Weiss, supra) and motion deblurring(see, e.g., Levin, A., Blind motion deblurring using image statistics,NIPS (2006)). Work that can be considered relevant to certain exemplaryembodiments of the present disclosure, such as that described in, e.g.,Levin, Fergus, Durand and Freeman, supra, and Joshi et al., Imagedeblurring and denoising using color priors, CVPR (2009), has beenapplied to non-blind deconvolution problems using IRLS in an attempt tosolve the deblurred image problem. Other types of sparse image priorsinclude, e.g., Gaussian Scale Mixtures (GSM) (see, e.g., Wainwright etal., Scale mixtures of Gaussians and the statistics of natural images,NIPS 855-861 (1999)), which have been used for image deblurring (see,e.g., Fergus et al., supra), denoising (see, e.g., Portilla et al.,supra) and student-T distributions for denoising (see, e.g., Welling etal., Learning sparse topographic representations with products ofstudent-t distributions, NIPS (2002) and Roth et al., supra). With theexception of Portilla et al., supra, these methods generally use CG andthus can be slow.

The alternating minimization procedure that can be adopted by certainexemplary embodiments in accordance with the present disclosure can be atechnique known as half-quadratic splitting (see, e.g., Geman andReynolds, Constrained restoration and recovery of discontinuities, PAMI14, 3, 367-383 (1992) and Geman and Yang, Nonlinear image recovery withhalf-quadratic regularization, PAMI 4, 932-946 (1995)). Recently, Wang,Y. et al., supra, showed that it could be used with a total-variation(TV) norm to deconvolve images. Exemplary embodiments according to thepresent disclosure can be considered to be related to this work: e.g.,certain exemplary embodiments can also use a half-quadraticminimization, but the per-pixel sub-problem is quite different. With theTV norm, the problem can be solved with a straightforward shrinkageoperation. As a consequence of using a sparse prior, the problem can benon-convex. Accordingly, solving the problem efficiently is one of theobjectives provided by exemplary embodiments according to the presentdisclosure.

Described in Chartrand, R., Fast algorithms for nonconvex compressivesensing: Mri reconstruction from very few data, IEEE InternationalSymposium on Biomedical Imaging (2009) and Chartrand and Staneva,Restricted isometry properties and nonconvex compressive sensing,Inverse Problems 24, 1-14 (2008), for example, is a non-convexcompressive sensing procedure, in which the usual l₁ norm on the signalto be recovered is replaced with a l_(p) quasi-norm, where p<1. Asplitting scheme can be used, resulting in a non-convex per-pixelsub-problem. To solve this, a Huber approximation (see, e.g., Chartrand,R., supra) to the quasi-norm can be used, which can allow for thederivation of a generalized shrinkage operator to solve the sub-problem.However, this approximates the original sub-problem, unlike exemplaryembodiments in accordance with the present disclosure.

SUMMARY OF EXEMPLARY EMBODIMENTS

At least one of the objects of various exemplary embodiments of thepresent disclosure is to overcome the deficiencies commonly associatedwith the prior art as discussed above, and provide exemplary embodimentsof computer-accessible medium, methods and systems for dark flashphotography.

For example, described herein is an exemplary embodiment of an apparatusfor providing illumination and obtaining an image, which can include,e.g., a first arrangement that can be configured to emit a flash oflight including ultra-violet light and/or infra-red light to illuminatea scene and/or one or more subjects, and a second arrangement that canbe configured to obtain an image of the illuminated scene and/or one ormore subject. A duration of time and/or an intensity level of the flashcan be selected so that wavelengths of the flash can be substantiallyinvisible to a human eye, for example. According to certain exemplaryembodiments of the present disclosure, at least, e.g., approximately 90%of the light can include ultra-violet light and/or infra-red light;while, in accordance with some exemplary embodiments, at least, e.g.,approximately 95% of the light can include ultra-violet light and/orinfra-red light.

It is possible that the visibility of the wavelengths of the flashperceived by the human eye can be between, e.g., approximately 200 timesless than the visibility of wavelengths of a standard flash of lighthaving substantially the same amount of energy as the flash. A standardflash can include, e.g., light having a substantially even distributionof wavelengths of between approximately 400 nanometers (nm) andapproximately 700 nm, where a majority of the light of the standardflash has a wavelength of between approximately 400 nm and approximately700 nm, for example.

Also described herein, for example, is an exemplary embodiment of anapparatus for providing illumination that includes an arrangementconfigured to emit light that can include ultra-violet light andinfra-red light. A majority of the light can be ultra-violet lightand/or infra-red light. According to certain exemplary embodiments, thearrangement can be further configured to emit near ultra-violet light,ultra-violet light having a wavelength of between approximately 360 nmand approximately 400 nm, and/or infra-red light having a wavelength ofbetween approximately 700 nm and approximately 800 nm, for example.

According to certain exemplary embodiments of the present disclosure, anexemplary apparatus can include a further arrangement configured toobtain an image. The image can be, e.g., of a scene and/or one or moresubjects illuminated by light including ultra-violet light and infra-redlight. A majority of the illuminating light can be ultra-violet lightand/or infra-red light. According to certain exemplary embodiments, themajority can include, e.g., at least approximately 90% of theilluminating light and/or at least approximately 95% of the illuminatinglight. According to some exemplary embodiments, the illuminating lightcan, e.g., consist of ultra-violet light and infra-red light.

In accordance with some exemplary embodiments of the present disclosure,the illuminating light can include, e.g., near ultra-violet light,ultra-violet light having a wavelength of between approximately 360 nmand approximately 400 nm, and/or infra-red light having a wavelength ofbetween approximately 700 nm and approximately 800 nm, for example.Additionally, according to some exemplary embodiments, the furtherarrangement can include attributes and be configured to emitultra-violet light and/or infra-red light having wavelengths that areselected based on the attributes.

Also described herein, for example, is an exemplary embodiment of aprocess for dark flash photography, which can include, e.g., obtaining afirst image of a scene and/or one or more subjects illuminated by lightincluding ambient light, obtaining a second image of the scene and/orone or more subjects illuminated with light including ultra-violet lightand infra-red light, and, using an apparatus, generating a third imagebased on the first image and the second image. According to someexemplary embodiments of the present disclosure, the process can furtherinclude displaying and/or storing the third image in a storagearrangement in a user-accessible format and/or a user-readable format,for example. The ambient and/or visible light can include light having awavelength of, e.g., between approximately 400 nm and approximately 700nm.

Additionally described herein, for example, is an exemplary embodimentof a computer-accessible medium having stored thereon computerexecutable instructions for dark flash photography. For example, whenthe executable instructions are executed by a processing arrangement,the processing arrangement can be configured to perform proceduresincluding, e.g., (a) obtaining at least two images, where a first imageof the images can be obtained using a dark flash illumination procedureand a second image of the images can be obtained using an ambientillumination, (b) determining a relationship between spectral bandscorresponding to the images, (c) identifying noise associated with thesecond image based on the relationship, and removing or reducing thenoise, and (d) generating a further image from the images with the noiseremoved or reduced, where the further image can have a higher qualitythan the first and/or second images. According to some exemplaryembodiments of the present disclosure, the relationships betweenspectral bands corresponding to the images can include one or morecorrelations between spectral bands corresponding to the images. Theprocessing arrangement can be further configured to, e.g., utilize anedge structure of the first image to remove or reduce the noise inaccordance with some exemplary embodiments of the present disclosure,for example.

Also described herein, for example, is an exemplary embodiment of aprocess for dark flash photography, including, e.g., (a) obtaining atleast two images, where a first image of the images can be obtainedusing a dark flash illumination procedure and a second image of theimages can be obtained using an ambient illumination, (b) determining arelationship between spectral bands corresponding to the images, (c)identifying noise associated with the second image based on therelationship, and removing or reducing the noise, and (d) generating afurther image from the images with the noise removed or reduced, wherethe further image can have a higher quality than the first and/or secondimages. According to some exemplary embodiments of the presentdisclosure, the process can further include displaying and/or storingthe third image in a storage arrangement in a user-accessible formatand/or a user-readable format, for example.

In addition, described herein, for example, is an exemplary embodimentof a system for dark flash photography, including a processingarrangement which, when executed, can be configured to e.g., (a) obtainat least two images, where a first image of the images can be obtainedusing a dark flash illumination procedure and a second image of theimages can be obtained using an ambient illumination; (b) determine arelationship between spectral bands corresponding to the images; (c)identify noise associated with the second image based on therelationship, and remove or reduce the noise; and (d) generate a furtherimage from the images with the noise removed or reduced, where thefurther image can have a higher quality than the first and/or secondimages.

Further described herein, for example, is an exemplary embodiment of acomputer-accessible medium having stored thereon computer executableinstructions for denoising and/or non-blind deconvolution of an output.For example, when the executable instructions are executed by aprocessing arrangement, the processing arrangement can be configured toperform procedures including, e.g., (a) obtaining at least one priorinformation that can have a hyper-Laplacian form, (b) selecting a set ofauxiliary variables that can include auxiliary variables correspondingto individual data points of the output, and (c) denoising and/ordeconvoluting at least a portion of the output using a continuationprocedure that can alternate between executing two sub-procedures insuccessive iterations to increase the strength of at least one parameterlinking the two sub-procedures in each iteration.

The continuation procedure can include, e.g., (i) using a firstsub-procedure, updating at least one of the data points of the outputwhile maintaining the auxiliary variables as constant, and (ii) using asecond sub-procedure, updating at least one of the auxiliary variableswhile maintaining the data points of the output as constant. Accordingto some exemplary embodiments of the present disclosure, the processingarrangement can be further configured, when executing the instructions,to, e.g., utilize a Fast Fourier transform to perform the firstsub-procedure. The processing arrangement can be further configured,when executing the instructions, to, e.g., utilize a lookup-table toperform the second sub-procedure in accordance with some exemplaryembodiments of the present disclosure. The lookup-table can include,e.g., precomputed values stored in a storage arrangement. The processingarrangement can be further configured, when executing the instructions,to, e.g., repeat the performance of the continuation procedure until theoutput is at least one of substantially denoised or substantiallydeconvoluted and/or denoised or deconvoluted in accordance withpredetermined criteria.

In accordance with some exemplary embodiments of the present disclosure,the output can be or include an image, and the prior information can beor include image prior information, for example.

These and other objects, features and advantages of the presentdisclosure will become apparent upon reading the following detaileddescription of exemplary embodiments of the invention, when taken inconjunction with the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Further objects, features and advantages of the invention will becomeapparent from the following detailed description taken in conjunctionwith the accompanying figures showing illustrative embodiments of theinvention, in which:

FIG. 1 is a set of exemplary images captured at a blur-free shutterspeed;

FIG. 2(a) is an exemplary graph illustrating exemplary spectral responsecurves for each of an exemplary camera's three color channels;

FIG. 2(b) is an exemplary graph illustrating irradiance from an theexemplary dark flash;

FIG. 2(c) is an exemplary graph illustrating exemplary spectrum receivedby the exemplary camera sensor when imaging a white surface illuminatedby the exemplary dark flash;

FIG. 3(a) is a photograph of an exemplary camera and dark flash system;

FIG. 3(b) is an illustration of an exemplary perceived brightness of anexemplary dark flash and a visible flash that provides a comparablecamera exposure;

FIG. 3(c) is an exemplary Macbeth color chart captured with a pair ofexemplary flash images, separated out into five spectral bands;

FIG. 4(a)-4(l) are exemplary graphs of exemplary spectral constraints inaccordance with the present disclosure;

FIG. 5 are exemplary illustrations of two portrait shots captured withan exemplary camera/flash under tungsten illumination;

FIG. 6 are exemplary illustrations of two different scenes captured withan exemplary camera/flash under fluorescent illumination;

FIG. 7 are exemplary images of close-ups of the example bottom group ofimages illustrated in the example of FIG. 6;

FIG. 8 are exemplary comparisons of an exemplary approach in accordancewith the present disclosure to different processing methods,illustrating two crops from the top group of images illustrated in FIG.6;

FIG. 9 are exemplary illustrations demonstrating how the value of a inexemplary Equation 5 can effect the reconstruction;

FIGS. 10(a) and 10(b) are exemplary graphs which are generated using adark/visible flash pair being able to accurately infer the spectralreflectance of objects;

FIG. 11 are exemplary images of the model in Equation 5 being used in avisible flash/no-flash setting;

FIG. 12 are exemplary close-ups of the scene of images illustrated inthe top group of FIG. 6 illuminated by candlelight;

FIG. 13 is an example graph of a UV exposure to human health hazard;

FIG. 14 are exemplary crops from two images being deconvolved by fourdifferent procedures;

FIG. 15(a) is an exemplary image of a typical real-world scene;

FIG. 15(b) is a graph of an exemplary empirical distribution ofgradients in the scene, along with a Gaussian fit, a Laplacian fit and ahyper-Laplacian;

FIG. 16 is a block diagram of a system in accordance with an exemplaryembodiment of the present disclosure; and

FIG. 17 is a block diagram of another exemplary embodiment of the systemin accordance with the present disclosure.

Throughout the figures, the same reference numerals and characters,unless otherwise stated, are used to denote like features, elements,components or portions of the illustrated embodiments. Moreover, whilethe subject invention will now be described in detail with reference tothe figures, it is done so in connection with the illustrativeembodiments. It is intended that changes and modifications can be madeto the described exemplary embodiments without departing from the truescope and spirit of the subject disclosure as defined by the appendedclaims.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

According to certain exemplary embodiments of the present disclosure,exemplary computer-accessible medium, methods and systems can beprovided for facilitating dark flash photography.

To overcome certain problems described above, disclosed and describedherein, for example, is an exemplary embodiment of an apparatus forproviding illumination and obtaining an image, which can include, e.g.,a first arrangement that can be configured to emit a flash of lightincluding ultra-violet light and/or infra-red light to illuminate ascene and/or one or more subjects, and a second arrangement that can beconfigured to obtain an image of the illuminated scene and/or one ormore subject. A duration of time and/or an intensity level of the flashcan be selected so that wavelengths of the flash can be substantiallyinvisible to a human eye while being able to be detected, received andprocessed by the exemplary second arrangement. An exemplary secondarrangement can be and/or include an exemplary camera sensor capable ofand/or configured to detect and receive light having wavelengths thatare substantially invisible to the human eye, e.g., outside of thevisible range between approximately 400 nanometers (nm) andapproximately 700 nm, for example. A majority of the light can beultra-violet light and/or infra-red light. According to certainexemplary embodiments, the majority can be a super-majority and/orinclude, e.g., at least approximately 80% of the light, at leastapproximately 90% of the light, at least approximately 95% of the lightand/or at least approximately 99% of the light, for example.

It is possible that the visibility of the wavelengths of the flashperceived by the human eye can be between, e.g., approximately 100 timesand approximately 300 times less than the visibility of wavelengths of astandard flash of light having substantially the same amount of energyas the flash. For example, according to some exemplary embodiments ofthe present disclosure, it is possible that the visibility of thewavelengths of the flash perceived by the human eye can be, e.g.,approximately 200 times less than the visibility of wavelengths of astandard flash of light having substantially the same amount of energyas the flash. According to other exemplary embodiments, it is possiblethat the visibility of the wavelengths of the flash perceived by thehuman eye can be, e.g., greater than approximately 300 times less thanthe visibility of wavelengths of a standard flash of light havingsubstantially the same amount of energy as the flash.

A standard flash can include, e.g., light having a substantially evendistribution of wavelengths of between approximately 400 nm andapproximately 700 nm, for example, where a majority of the light of thestandard flash has a wavelength of between approximately 400 nm andapproximately 700 nm. The majority can be a super-majority and/orinclude, e.g., at least approximately 80% of the light, at leastapproximately 90% of the light, at least approximately 95% of the lightand/or at least approximately 99% of the light, for example.

For example, according to exemplary embodiments of the presentdisclosure, a pair of images can be acquired in the manner offlash/no-flash photography, one using the dark flash and the secondusing ambient illumination alone. For the latter to be blur-free, a fastshutter speed can be used, resulting in high noise levels in dim light(e.g., low light level conditions). If the non-visible and visiblechannels are close in wavelength, one or more strong relationships canexist between them. Such relationship(s) can be and/or include, e.g., acorrelation, coherence, mutual information, common components, etc.Disclosed and described herein is, e.g., a novel type of constraint thatcan exploit the relationships between spectral bands, such as, thecorrelation between spectral bands. For example, using this exemplaryconstraint, the edge structure of the dark flash image can be used toremove the noise from the ambient image, yielding a high quality resultthat lacks the shadow and specularity artifacts that can typically bepresent in the flash image. An exemplary procedure for exploiting therelationships between images (and/or the spectral bands thereof) caninclude, for example, one or more procedures which can be implementedusing one or more computational methods and systems, such as, e.g.,correlation, coherence, mutual information, principal componentsanalysis, independent components analysis, canonical correlation,general linear model, etc.

Also disclosed and described herein are exemplary procedures of howexemplary camera and flash hardware and spectral constraints can be usedin a range of additional applications, including, e.g., inferringspectral reflectance functions of materials in the scene and denoisingindividual color channels of images captured with standard cameras.

Further disclosed and described herein is an exemplary deconvolutionapproach that is several orders of magnitude faster than existingtechniques that use hyper-Laplacian priors. An alternating minimizationprocedure can be used where one of the two phases is a non-convexproblem that is separable over pixels. This per-pixel sub-problem can besolved with a lookup table (LUT). Alternatively, for two specific valuesof α, ½ and ⅔, an analytic solution can be found by finding the roots ofa cubic and quartic polynomial, respectively. According to exemplaryembodiments of the present disclosure (using, e.g., either LUTs oranalytic formulae) it is possible to deconvolve a 1 megapixel image inless than approximately 3 seconds, achieving comparable quality toexisting methods such as iteratively reweighted least squares (IRLS)that can take approximately 20 minutes to complete. Furthermore,exemplary procedures in accordance with the exemplary embodiments of thepresent disclosure can be relatively general and thus easily be extendedto related image processing problems beyond the deconvolutionapplication described herein for example.

Efficient exemplary procedures are also disclosed herein for non-blinddeconvolution of images using a hyper-Laplacian image prior for 0<α≦1.Exemplary algorithm uses an alternating minimization scheme where thenon-convex part of the problem is solved in one phase, followed by aquadratic phase which can be efficiently solved in the frequency domainusing FFTs. Exemplary embodiments according to the present disclosurecan focus on, e.g., the first phase where at each pixel it is possibleto solve a non-convex separable minimization. Two exemplary approachesto solving this sub-problem are provided. The first uses a lookup table(LUT); the second is an analytic approach specific to two values of α.For α=½ the global minima can be determined by finding the roots of acubic polynomial analytically. In the α=⅔ case, the polynomial is aquartic whose roots can also be found efficiently in closed-form. BothIRLS and exemplary embodiments according to the present disclosure cansolve a series of approximations to the original problem. However, inexemplary methods according to the present disclosure, eachapproximation can be solved by alternating between the two phases abovea few times, thus avoiding the expensive CG descent used by IRLS. Thisallows exemplary embodiments according to the present disclosure tooperate several orders of magnitude faster. Although exemplaryembodiments according to the present disclosure focus on the problem ofnon-blind deconvolution, in view of the teaches disclosed herein, itwould be straightforward to having ordinary skill in the art to adaptcertain exemplary embodiments according to the present disclosure toother related problems, such as, e.g., denoising or super-resolution.

As described herein, a pair of images can be received that are taken attwo points in time, e.g., t1 and t2. For example, a pair of images canbe captured in relatively quick succession. The first image can be takenwith the dark flash on, and the exposure can be of a duration which canbe relatively short, e.g., 1/40 seconds or less. The second image can betaken right after the first without the dark flash, for example. In thecase of low ambient light levels in the scene, such an image may be,e.g., noisy and blurry. The level of noise and blur may depend on, e.g.,the exposure time of this image. As the exposure time is increased, theimage may become more blurry, and may become less noisy. Decreasing theexposure time can have the opposite effect: the blur can be reduced, butnoise levels may be increased. The blurring in the ambient image can bedue to, e.g., the motion of the camera during the long exposure. Thisblurring function, which can also be called the blur kernel, can beunknown.

According to some exemplary embodiments in accordance with the presentdisclosure, the unknown blur kernel can be estimated, and further theunblurred version of the ambient image can be estimated. According tosome exemplary embodiments of the present disclosure, this blinddeblurring problem can be solved in two exemplary procedures. In thefollowing example, the flash image can be referred to as F and theambient (non-flash) image is referred to as A.

The blurring in A, if any, can be assumed to be caused, e.g., due tocamera shake. The blur kernel k can be estimated in a number of ways,for example. One exemplary way is to estimate k by solving the problemdescribed below. In this example, F_(j) and A_(j) refer to an individualchannel j, where j can be, e.g., 1, 2, 3, for red, green and bluechannels respectively:

$\begin{matrix}{{\arg {\min\limits_{k}\; {\lambda {{{F_{j} \oplus k} - A_{j}}}^{2}}}} + {k}^{2}} & (1)\end{matrix}$

where ⊕ is the convolution operator, and λ is a regularizationparameter. Another exemplary procedure and/or method to estimate theblur kernel is to solve the following exemplary problem:

$\begin{matrix}{{\arg {\min\limits_{k}\; {\lambda {{{{\nabla F_{j}} \oplus k} - {\nabla A_{j}}}}^{2}}}} + {k}^{2}} & (2)\end{matrix}$

where ∇ refers to the gradient of the image. Following the estimation ofk, the estimation of k can be normalized so that, e.g., the entries of ksum to 1. Any of the three exemplary channels j can be used for theestimation of k. According to some exemplary embodiments, it may bepreferable to use the exemplary green channel (e.g., j=2).

When k is estimated, A can be deblurred and denoised to create asubstantially noise-free and blur-free image with the substantiallycorrect colors. The gradient information from the clean flash image Fcan also be used according to some exemplary embodiments of the presentdisclosure. The reconstructed image can be called, e.g., R. Threesubstantially clean channels R_(j) can be recovered by utilizing thecolor information in A_(j) and the gradient information in F_(j).According to certain exemplary embodiments of the present disclosure,the following exemplary optimization problem can be solved:

$\begin{matrix}{{\arg {\min\limits_{R_{j}}{\sum\limits_{p}^{\;}\; {\lambda \left( {{\left( {R_{j} \oplus k} \right)(p)} - {A_{j}(p)}} \right)}^{2}}}} + {\kappa {{R_{j}(p)}}^{\alpha}} + {{{\nabla{R_{j}(p)}} - {\nabla{F_{1}(p)}}}}^{\alpha} + {{{\nabla{R_{j}(p)}} - {\nabla{F_{3}(p)}}}}^{\alpha}} & (3)\end{matrix}$

where p refers to the pixels in the channel, and can usually be ≦1. Inthis example, k represents the discovered blur kernel from the previousstep. This exemplary optimization problem can be solved with thesubstantially fast method as described herein below.

Currently, most flashes can be considered to be relatively fast, e.g.,in the range of 1/500 seconds to 1/1000 seconds. To provide sufficientand/or enough illumination to a scene, the flash power typically can beselected to be high enough so that significant energy can be deliveredin this relatively short time frame. To make the dark flash lessdazzling, according to certain exemplary embodiments of the presentdisclosure, the duration of the flash can be reduced (e.g., slowed down)to approximately 1/100 of a second, with a corresponding decrease in theflash power if preferred.

As described herein, an exemplary fast numerical procedure, exemplarymethod and/or scheme is disclosed that can be used to solve problems ofthe type associated with the exemplary optimization problem describedabove, which can be nonconvex when, e.g., α<1 and can be typically hard(e.g., difficult and/or slow) to solve. Described herein is an exemplarysplitting procedure and/or scheme that can reformulate the exemplaryoptimization problem described above by exemplary Equation 3 into twoexemplary subproblems, which can be called, e.g., the “w sub-problem”and the “u sub-problem” (see, e.g., Chartrand, R., supra). As describedherein, the exemplary w sub-problem can be solved analytically forcertain values of the exemplary exponent α in the exemplary optimizationproblem described above, specifically, e.g., α=½ and α=⅔. Numerically,according to certain exemplary embodiments, an exemplary Look Up Table(LUT) can be used that can directly encode the exemplary substantiallyoptimal solution of the exemplary w sub-problem for a wide range ofexemplary values, and use, e.g., linear interpolation to determine theexemplary solution for entries that are not in the LUT. This exemplaryprocedure and/or scheme can be effective because it can be knowna-priori the range of exemplary values for the different variables inthe exemplary w sub-problem. The exemplary LUT-based procedure can berelatively very fast as compared to other potential related procedures,and can enable and/or allow the exemplary optimization problem describedabove for any a value to be solved.

FIG. 1, for example, shows certain exemplary images 101, 102 that werecaptured at a blur-free shutter speed. Image 101 was captured using amulti-spectral flash (F), while image 102 using ambient illumination (A)which in this exemplary case is 1/100th of that which can be typicallyassociated for a correct and/or optimal exposure. Images 101 and 102were combined to provide an output image 103 (R), which can be ofcomparable quality to a reference long exposure show, as shown by image104 (L). As can be seen in FIG. 1, an exemplary camera and flash systemin accordance with the present disclosure can provide dazzle-freephotography by, e.g., substantially or completely hiding the flash inthe non-visible spectrum.

Exemplary Dark Flash Hardware

In accordance with certain exemplary embodiments of the presentdisclosure, the images can be captured, e.g., one with an exemplary darkflash (F) and another using ambient lighting alone (A). The pixel valuep in channel j of image F can depend on three terms: e.g., the spectralresponse of each camera channel C_(j)(λ) at wavelength λ, theillumination spectrum of the dark flash I^(f)(λ), and the surfacereflectance function S(p, λ) at the point in the scene. These cancombine in a linear fashion:

F _(j)(p)∫C _(j)(λ)I ^(f)(λ)S(p,λ)dλ  (4)

with j={1, 2, 3} being the index of the camera channel. Evenillumination can be presumed (e.g., I_(f)(λ) does not depend on p). Theambient image A can be formed in a similar fashion, using illuminationI^(α)(λ) which can scale with the exposure interval. A₁, A₂ and A₃ canrecord red, green and blue wavelengths, respectively, under typicalillumination. Through the choice of flash and camera, I^(f)(λ) and thechannel sensitivities C_(j)(λ) can be controlled.

In accordance with certain exemplary embodiments of the presentdisclosure, it can be preferable to use off-the-shelf consumer hardwarewhere possible to minimize the costs of the system and make itrelatively easily reproducible. For example, a camera that can be usedis, e.g., the Fuji IS Pro, which has been marketed for applicationsinvolving UV and IR since it lacks an IR sensor filter. A flash can be,e.g., the Nikon SB-14UV. Both the camera and the flash can be equippedwith carefully selected filters (as described herein below, for example)that shape both I^(f)(λ) and C_(j)(λ) in accordance with some exemplaryembodiments of the present disclosure. These filters can remain in placefor both shots, thus the pair of images can be taken in quicksuccession, limited by the 3 frames/second rate of the camera, forexample. According to certain exemplary embodiments, an IR-block filtercovering the sensor can be removed temporarily by, e.g., utilizing asliding mechanism. The flash can be used at full power for all shots,with the cycle time being sufficiently long that it does not fire forthe second shot, giving an image with ambient illumination alone.Systems in accordance with exemplary embodiments of the presentdisclosure can be no more complex to operate than, e.g., a standardDSLR.

An exemplary form of I^(f)(λ) and how it can be recorded by a camerawhile remaining largely invisible to humans in accordance with thepresent disclosure is described with reference to FIG. 2.

FIG. 2(a) shows an exemplary graph illustrating spectral response curvesC_(j)(λ), j={1, 2, 3} for each of an exemplary camera's three colorchannels, as depicted by curves 211, 212, 213. As shown, with no IRsensor filter, the responses can extend considerably beyond the visiblerange 202 of approximately 400-700 nm.

FIG. 2(b) shows an exemplary graph illustrating irradiance 1 m from thedark flash I^(f)(λ). The spectrum of the exemplary dark flash I^(f)(λ)221 is shown to have two distinct emission lobes 222, 223, both justoutside the visible range 202. The first 222, consisting of UV light,couples with the small part of channel j=3's response extending belowapproximately 400 nm in region 201. The second lobe 223 in the IR region203 between approximately 700 and 800 nm is picked up by channel j=1which can respond relatively strongly. Thus, certain exemplaryembodiments of dark flash can allow the recording of two independentmeasurements (e.g., 222, 223) at each location in a scene within asingle image: one in UV 201 recorded in F₃, the other in IR 203 recordedin F₁.

FIG. 2(c) shows an exemplary graph illustrating exemplary spectrumreceived by the exemplary camera sensor when imaging a white surface(S(p, λ)=1) illuminated by the dark flash. The curves 231, 232, 233 arethe product of those shown in Figures (a) and (b). The recorded pixelvalues for the three channels are the integrals of these curves, ascalculated in Equation 4, described herein below). As can be seen, underthe exemplary dark flash embodiment: no channel recorded in the visiblerange 202. Rather, as illustrated by curve 233, channel j=3 measured inthe UV range and, as illustrated by curve 231, channel j=1 responded toIR wavelengths.

In accordance with some exemplary embodiments of the present disclosure,the flash/no-flash images can capture the scene at 5 different spectralbands, assuming the ambient illumination is dim compared to the outputof the flash: 1. UV (approximately 370-400 nm) in F₃; 2. Blue(approximately 400-500 nm) in A₃; 3. Green (approximately 500-600 nm) inA₂; 4. Red (approximately 600-700 nm) in A₁ and 5. IR (approximately700-800 nm), recorded in F₁.

FIG. 3(a) shows an exemplary photo of an exemplary camera 311 and darkflash system 312 in accordance with the present disclosure. FIG. 3(b)illustrates the perceived brightness of an exemplary dark flash 321 anda visible flash 322 that provides a comparable camera exposure. Tocapture them in a single image, the visible flash was attenuated by afactor of approximately 220 using exemplary neutral density filters.Without these filters, the dark flash may not be visible in anon-saturated 8-bit image. FIG. 3(c) is an exemplary Macbeth color chartcaptured with a pair of exemplary flash images (visible 331 and dark332), separated out into the five spectral bands described above. See,e.g., UV 341 (approximately 370-400 nm) in F₃, Blue 342 (approximately400-500 nm) in A₃, Green 343 (approximately 500-600 nm) in A₂, Red 344(approximately 600-700 nm) in A₁ and IR 345 (approximately 700-800 nm)recorded in F₁. Subplot 346 shows the UV band with a UV-block filterattached to the camera that has a relatively sharp cut-off atapproximately 400 nm. The low intensities in this band show that theexemplary camera is genuinely recording UV light, not blue light fromfluorescence caused by the UV part of the flash, for example.

For comparison purposes, a standard visible flash whose power can beadjusted to give comparable camera exposure to the dark flash was used.FIG. 3(b) shows a relative perceived brightness of the dark and visibleflashes by capturing them using a standard DSLR whose spectral responsecan be close to that of, e.g., human eyes (thus the brightness in theimage generally corresponds to human perception). A quantitativeanalysis of their relative brightness is described further herein below.

With respect to safety issues, referring back to FIG. 2(b), exemplaryembodiments of dark flash can emit energy just outside the visiblewavelength range, centered approximately 380 nm with negligible energybelow approximately 360 nm or above approximately 400 nm (until theexemplary IR lobe at approximately 700 nm). The potential health hazardposed by UV light can strongly depend on the wavelength. For example,those close to the visible range (e.g., approximately 400 nm) aregenerally orders of magnitude safer than the shorter wavelengthcomponents of sunlight. Exemplary embodiments of dark flash inaccordance with the present disclosure can be relatively very close tothe visible range, even closer than, e.g., blacklights that can be foundin public establishments such as bars and nightclubs, for example, whichblacklights can have a broader spectral width centered at approximately360 nm. In the United States of America, acknowledged regulationsregarding the safe daily exposure to UV light are provided in theThreshold Limit Values (TLV) booklet, published by the ACGIH (TLVs2001).

According to certain exemplary embodiments of the present disclosure,the absolute spectral irradiance of the exemplary flash can be carefullymeasured using a spectrometer. Using the TLV tables, the approximatemaximum safe number of flashes per day can be computed, which can be,e.g., approximately 130,000 at approximately 1 m from the flash. Forexample, if approximately 30 minutes outside in the sun results in themaximum permissible UV dose on a bright summer day, then each flash canbe equivalent to being outside for approximately 1/100th second. Thus,exemplary embodiments of dark flash can likely pose no significantsafety hazard. Further details of these calculations are provided hereinbelow.

Exemplary Dark Flash Processing

An exemplary images, F and A can be captured using a shutter speedsufficient to avoid camera shake. Assuming that the ambient illuminationis weak, A will typically be very noisy and the illumination in F willbe dominated by the dark flash I^(f)(λ). An exemplary image R can besought having edges that are close to those in F and whose intensitiesare close to a denoised version of A, which should be similar to along-exposure shot of the exemplary scene L, for example.

Standard approaches to denoising typically use spatial priors that canenforce sparsity on image gradients (see, e.g., Portilla et al., supra).In the flash/no flash scenario, F can contain high-frequency detailsthat can assist the denoising process. But unlike conventionalflash/no-flash photography, exemplary dark flash and ambientilluminations I^(f)(λ) and I_(α)(λ) are virtually non-overlapping, thusthe colors in F can be quite different to those in the ambient image Aor the long-exposure L. In accordance with exemplary embodiments of thepresent disclosure, a solution that uses the strong correlations betweencolor channels as a constraint in an optimization scheme which computesR from A and F can be used.

Exemplary Spectral Constraints

FIGS. 4(a)-4(l) show 1-D exemplary graphs of the spectral constraints inaccordance with the present disclosure, using a scan line across 3squares in the color chart of FIG. 3. The 1-D example in FIG. 4 shows ascanline across 3 squares in the exemplary color chart from FIG. 3. FIG.4(a) shows the intensities from the red channel 411 of a long exposureshot (L₁) and IR 412 from an exemplary embodiment of the dark flash(F₁). Although the intensities are shown as being different, the edgescan be aligned since the spectral reflectance at red and IR wavelengthscan be correlated with one another. The alignment of the edges isapparent in FIG. 4(b) where the exemplary gradients 421, 422 along thescanline ∇F₁ and ∇L₁ are shown (∇F₁(p)=F₁(p)−F₁(p−1)), the differencebetween adjacent pixels p). As can be seen, this gradient signal isrelatively sparse, being close to zero everywhere but in a fewlocations.

FIG. 4(c) shows a curve 431 of the difference between the two exemplarygradient signals 421, 422 ∇F₁−∇L₁, which is also sparse, as shown byshape of the histogram 441 in FIG. 4(d). In considering a dark flash andnoisy ambient image pair together, as shown in FIG. 4(e)-(h), forexample, the difference between gradients ∇F₁−∇A₁ (as shown by curve 451in FIG. 4(g)) may no longer be sparse. This is also shown by thecorresponding Gaussian-shaped histogram 461 in FIG. 4(h), for example.

Reflecting the sparse distribution of ∇F₁−∇L₁ in FIG. 4(d), exemplaryspectral constraints can take the form of a sparse norm on the gradientdifference between channels in the reconstructed image R and the flashimage F₁, e.g., |∇R_(j)−∇f₁|^(α) where α≦1. This exemplary embodimentcan encourage the edge structures in R_(j) to align spatially with thosein F₁ while allowing their magnitudes to differ. Thus, whentransitioning between two materials, it can be possible that it does notmatter if the spectral reflectances are different in visible and IR/UVbands, provided that there is a significant edge in IR/UV. If an l₂ normis used, this may not be the case, and ΔR_(j) and ∇F₁ can closely match,even at material transitions, causing artifacts in R_(j). While aconventional spatial prior, such as |∇R_(j)|^(α), α<1, can also reducenoise, it can not encourage the edges to align with those of F which areclose to those of the desired solution L as can be accomplished withvarious exemplary embodiments in accordance with the present disclosure.

An similar exemplary constraint can also be imposed to the UV channel:|∇R_(j)−∇F₃|^(α), where F₃ records UV and F₁ records IR. For R₃ (theblue channel), this can be a strong constraint since, in terms ofwavelength, blue is much closer to UV than to IR. In this example, 1-Dgradients have been considered, but both x and y gradients can be usedin real-world problems, with separate terms for each. As used herein, ∇refers to both ∇_(x) and ∇_(y).

Exemplary Spatial-Spectral Cost Function

An exemplary cost function in accordance with the present disclosure canconsist of, e.g., three main terms: (i) Likelihood: the intensities ofthe reconstruction R_(j) should be close to those of the noisy ambientimage A under an l₂ norm, assuming a Gaussian noise model; (ii) Spatialprior: ∇R_(j) should be small under a sparse norm, reflecting theheavy-tailed nature of image gradients, which spatial prior term canhelp, e.g., to provide a further boost to image quality; and (iii)Spectral constraint: ∇R_(j) can be close to both ∇F₁ (IR) and ∇F₃ (UV)under a sparse norm, as described herein above, for example.

As with existing flash/no-flash techniques, exemplary embodiments inaccordance with the present disclosure can use a shadow and specularitymask m(p) which can, e.g., remove artifacts from the flash image.Details of the mask construction are described further herein below. Forexample, the overall cost function for each channel j can be:

$\begin{matrix}{\underset{R_{j}}{argmin}{\sum\limits_{p}\; \left\lbrack {\underset{\underset{Likelihood}{}}{\mu_{j}{m(p)}\left( {{R_{j}(p)} - {A_{j}(p)}} \right)^{2}} + \underset{\underset{Spatial}{}}{\kappa \; {m(p)}{{\nabla{R_{j}(p)}}}^{\alpha}} + \underset{\underset{{IR}\mspace{14mu} {Spectral}}{}}{{{{\nabla{R_{j}(p)}} - {\nabla{F_{1}(p)}}}}^{\alpha}} + \underset{\underset{{UV}\mspace{14mu} {Spectral}}{}}{{{{\nabla{R_{j}(p)}} - {\nabla{F_{3}(p)}}}}^{\alpha}}} \right\rbrack}} & (5)\end{matrix}$

As used herein, unless otherwise noted, κ=1 and α=0.7. Each channel jcan be solved separately. For example, m(p) can have the effect ofincreasing the weight on the likelihood and spatial terms in regions ofshadows or specularities. It can also be assumed that the UV and IRspectral terms have equal weight for all channels j. Hence, theweighting on the reconstruction term for each channel p is the mostimportant parameter in this exemplary model and can strongly depend onthe noise level of the ambient image A. Since the blue channel can oftenbe significantly noisier than the other channels, a different value canbe used for μ₃ than for μ₁ and μ₂, which can be set to be the same. Ifμ_(j) is set to a large value then the colors of R can be close to thoseof A at the expense of increased noise. Conversely, if μ_(j) is smallthen the noise in R can be reduced, but the colors can deviate fromthose in A. Choosing the value of μ_(j) can be done semi-automaticallyfrom the level of under-exposure of A (given by a camera's exposuremeter) and the camera's ISO setting, for example. According to certainexemplary embodiments, the value can also be fine-tuned on a small imagepatch before processing the entire image. Typical values can range from,e.g., μ_(j)=approximately 5 (high noise) to μ_(j)=approximately 40 (lownoise).

Returning to the 1-D example of FIGS. 4(a)-4(l), the scanline across thecolor chart for exemplary reconstructed image R in FIGS. 4(i)-4(l) isshown. Despite the spectral reflectances of the squares being quitedifferent, the intensities of R₁ shown by curve 471 in FIG. 4(i) closelymatch those of the desired solution L₁ in FIG. 4(a). As shown, R₁ can bekept close to A₁ (e.g., shown as curve 481 in FIG. 4(e)) by an exemplarylikelihood term, while the sparse norm on the spectral terms can removethe noise.

Equation 5 (which is non-convex if, e.g., α=0.7) can be optimized using,e.g., Iterative Re-weighted Least Squares (see, e.g., Levin, Fergus,Durand and Freeman, supra), initializing with R_(j)=F_(j). Due torelatively poor conditioning of the least-squares systems, an incompleteCholesky preconditioner can be used to speed convergence.

For example, for a 1.3 megapixel image, an unoptimized Matlabimplementation can takes approximately 25 minutes for all 3 channels,with 5 iterations per channel. As this may be unacceptably slow for somepractical situations, a considerable speedup can be achieved by, e.g.,setting α=1. This can make the problem convex and fast numerical schemescan be used (see, e.g., Wang, Y. et al., supra), that can result in aprocessing time of approximately 3 minutes, comparable to relativelyefficient implementations of the cross-bilateral filter. However, someimage quality can be lost in using α=1, as described herein below withreference to FIG. 9, for example.

Exemplary Pre- and Post-Processing

Pre-Processing.

The exemplary images shown and described herein were captured in RAWmode. They were then demosaiced and manually white-balanced using someneutral-colored object (e.g. a wall or calibration target) in the scene.The mask m(p) was built using a similar procedure to that described in,e.g., Petschnigg et al., supra. The shadows were detected by findingareas where |F−A| is relatively very small. Specularities were found bylooking for pixels saturated in F₁ (IR channel). In areas ofshadow/specularity m(p)=5 and m(p)=1 in the other areas, smoothlyvarying between the two at the boundaries. In high noise conditions, itis possible to apply a small Gaussian smoothing to A₁ to break up anyspurious image structure formed by the noise. The optimization can thenbe performed on the linear tonescale images (e.g., without gammacorrection).

Post-Processing.

If the ambient light levels are low, the colors in the ambient image canbecome imbalanced, particularly with a blue tint due to excessive noiselevels in the blue channel, for example. Thus, the output of theoptimization can also have a similar color cast and generally will notlook similar to a long-exposure shot L. To compensate for this, anadditional color correction operation that applies a global colormapping to R can be used, for example. To generate this mappingfunction, the tone response curve of an exemplary camera can bedetermined for each color channel using a stack of images taken over awide range of exposures (see, e.g., Debevec and Malik, Recovering highdynamic range radiance maps from photographs, ACM Transactions onGraphics (Proc. SIGGRAPH) 31, 3, 369-378 (1997)). Particular care can betaken when fitting the parametric model to the low intensity part of thecurve. In this regime, the sensor noise can cause the curve to benon-linear, in turn giving rise to the color casts observed in verynoisy images. By passing each R_(j) through its appropriate mappingfunction, it is possible to infer the true value of each pixel, yieldingcolors close to those in a long-exposure shot L. Finally, it is possiblegamma-correct the images for display, using γ=1.8, for example.

Exemplary Results

For an exemplary dark flash system to be practical in accordance withcertain exemplary embodiments of the present disclosure, it can bepreferable to achieve high quality reconstructions in low levels ofambient illumination.

For example, FIG. 5 shows exemplary illustrations of two portrait shotscaptured with an exemplary camera/flash under tungsten illumination.Within each group, column 1 510 shows the dark flash shot (F) 511, 513and long exposure reference (L) 512, 514. The exemplary results areshown in Columns 2 520, 3 530 and 4 540. For each ambient image (A) ofdecreasing exposure (yielding increased noise), the reconstructed output(R) is shown. Column 5 550 shows a visible flash image (V) 551, 553,along with a visible flash shot (D) 552, 554 attenuated with neutraldensity filters so that it is comparably dazzling to F. The Low, Mediumand High noise levels correspond to 6, 7 and 8 stops of underexposurerespectively (e.g., corresponding to 1/64th, 1/128th and 1/256th ofambient long exposure). In the lower group, corresponding to shot 513and exposure reference 514, a zoomed-in section is shown, separated intored, green, blue color channels 561, 562 and 563, respectively.

FIG. 6 shows exemplary illustrations of two different scenes 601, 602captured with an exemplary camera/flash under fluorescent illumination.Within each group 610, 620, row 1 611, 621 and row 2 612, 622 show shotsunder ambient illumination (A) of decreasing exposure (yieldingincreased noise) and exemplary reconstructed output (R). Row 3 613, 623show, from left to right: Long exposure reference (L) 631, 641, Visibleflash shot (V) 632, 642 and dark flash shot (F) 633, 643. In the topgroup 610, Low, Medium and High noise levels correspond to 5, 6 and 7stops of underexposure respectively (equating to 1/32nd, 1/64th and1/128th of ambient long exposure). In the bottom group 620, Low=5.5,Medium=6.5 and High=7.5 stops underexposed (corresponding to 1/45th,1/90th and 1/180th of ambient).

The test images shown in FIGS. 5 and 6 were captured using two differenttypes of ambient illumination (tungsten and compact fluorescent) andcontain a wide range of materials and colors. As described above, toshow how the noise levels vary across color channel, a small region intwo of the exemplary images shown in FIG. 5 is also shown, separated outinto its constituent color planes. This typically reveals the bluechannel to be far noisier than the others.

To make comparisons straightforward, the shutter speed used to capturethe flash/no-flash pair can be varied, thus simulating different levelsof ambient illumination. In practice, according to some exemplaryembodiments of the present disclosure, the shutter speed can be set tothe slowest level that avoids camera shake, irrespective of the level ofambient light. As the light levels drop, the ambient image can becomenoisier (while the dark flash image F can stay constant) thus making thereconstruction harder. Three different noise scenarios are describedherein: (i) Low, where it is possible to achieve reconstructions closeto a long exposure reference shot; (ii) Medium, where the reconstructionis acceptable in terms of quality and (iii) High, where a significantdegradation in quality is visible and the failure modes of the algorithmare evident. At each noise level, the degree of under-exposure of theambient image A, relative to the long exposure reference L, is quoted.These can range from 1/32nd of ambient illumination (e.g., as shown inthe exemplary images of FIG. 6, group 610), down to 1/256th for theexemplary portrait shots shown in FIG. 5, for example. Assuming 1/30thof a second can be used to avoid camera shake, the results can beequivalent to taking pictures in conditions where exposures ranging from1 second to 8 seconds could otherwise be required. Techniques thatpermit relatively blurfree photography at slow shutter speeds, such asimage stabilizers, could extend the range of operation of some exemplarydark flash systems to even longer equivalent exposures.

Achieving accurate alignment between F and A can be an importantpractical issue due to the spectral constraints. While various softwareapproaches for image registration exist (see, e.g. Baker et al.,Lucas-kanade 20 years on: A unifying framework, International Journal ofComputer Vision 56, 221-255 (2004)), a commercial implementation of anexemplary system could use a hardware approach based on sensors that cancapture the exemplary images with very little to virtually no delaybetween them (see, e.g. specifications of Fuji Finepix Z10fd), providingfor good alignment. Thus, according to certain exemplary embodiments ofthe present disclosure, it is possible to resolve this issue and capturethe shots using such hardware with a tripod, for example.

At high noise levels, some color deviations and loss of detail can beobserved. This can be a consequence of low μ_(j) values which give thelikelihood term little weight in the optimization. At all noise levels,it is possible that reconstructions can contain some artifacts thatresult from the dark flash illumination. If a material absorbs both UVand IR strongly, then F can contain no gradients to guide thereconstruction. Examples of this include the freckles on the man shownin FIGS. 1 and 5, and the red lips of the doll shown in FIG. 6. This canbe relatively uncommon, as demonstrated by the range of colors andmaterials in the exemplary shots, the vast majority of which can beaccurately recovered. In particular, human skin and hair, two materialsthat can be relevant to exemplary dark flash applications, can beplausibly reproduced.

Exemplary Comparison Experiments

FIG. 7 shows exemplary close-up images of the bottom group 620 of FIG.6. FIG. 7 illustrates the importance of having UV and IR in someexemplary embodiments of dark flash by showing the results when thecorresponding spectral term in the cost function of Equation 5 isremoved, and thus the both spectral terms in Equation 5 being used.Panel 701 shows the blue channel of reconstructed image R using both theUV and IR spectral terms. Panel 702 shows the blue channel using onlythe IR spectral term. Panel 603 shows the red channel of reconstructedimage R using both the UV and IR spectral terms. Panel 704 shows the redchannel using only UV spectral term. As shown, the removal of the flashin the adjacent band can cause a degraded result. Thus, FIG. 7 showsboth the UV and IR components being used, since, if either were to beremoved, the adjacent spectral bands (blue and red, respectively) in Rcan become degraded.

FIG. 8 shows exemplary comparisons of an exemplary approach inaccordance with the present disclosure to different processing methods,showing two crops from group 610 of FIG. 6, along with the blue channelof the first crop. The top set 810 uses a dark flash/ambient image pair,while the bottom set 820 uses the ambient image only. Image R 811 wasreconstructed using spectral constraints. Image CB 812 is of a pipelineas described in, e.g., Petschnigg et al., supra, based oncross-bilateral filter and detail enhancement. Image B 821 was producedusing a bilateral filter of ambient image (see, e.g., Tomasi et al.,supra). Image NN 822 was produced using the Noise Ninja commercialdenoising plugin for Photoshop (see, e.g., Christian and Zapata, NoiseNinja, Photoshop denoising plugin, available athttp://www.picturecode.com (last accessed Jan. 7, 2010)). As shown inFIG. 7, for example, the exemplary reconstruction approach in accordancewith the present disclosure produced superior results to the examplecross-bilateral approach and the example standard denoising methods.

As shown in FIG. 8, the mid-noise case is used for comparison of anexemplary embodiment in accordance with the present disclosure toalternate methods. As shown, using the dark flash/ambient image pairwith the processing pipeline based on the cross-bilateral filter anddetail enhancement approach described in, e.g., Petschnigg et al.,supra, the results obtained were inferior to results obtained throughusing exemplary approaches in accordance with the present disclosure.The range term in the cross-bilateral filter caused the edge strength inthe flash image F to directly influence the smoothing of the ambientimage A. Thus it can only operate correctly if the edges in F and A areclosely matched in magnitude, an unrealistic assumption since spectralreflectances typically differ between bands. In contrast, an exemplarymodel in accordance with the present disclosure can permit the edgemagnitudes to differ when α≦1 in Equation 5, giving a reconstruction ofsuperior quality. Second, two approaches that have been attempted todirectly denoise the ambient image include: (i) bilateral filtering(see, e.g., Tomasi et al., supra) and (ii) a commercial denoising tool,Noise Ninja (see, e.g., Christian and Zapata, supra). Both methods weredemonstrated to have performed poorly in comparison to theflash/no-flash approaches.

FIG. 9 shows exemplary illustrations demonstrating how the value of α inEquation 5 can effect the reconstruction. When a non-sparse norm is used(α=2), as shown in row 903, the ambient colors can bleed. This can beprevented by, e.g., using α≦1, as shown in row 902, with someimprovement in quality for α=0.7, as shown in row 901.

As shown in rows 901, 902, for values ≦1, R can contain crisp edges,even if the spectral reflectances of the materials in visible andnon-visible wavelengths differ somewhat, as can typically be the case.Setting α=2 has the undesirable effect of causing the colors to bleedbetween regions. When α=2 the spectral constraints force the edges inthe UV/IR flash and ambient to be the same, an unrealistic assumptiongiven that they are captured at different wavelengths.

Exemplary Fluorescence

Certain materials can fluoresce when illuminated by the UV component ofan exemplary flash, the most common instances being white items ofclothing such as the stripes shown in FIG. 5 (top portrait).Fluorescence can manifest itself as visible blue light that gives anunnaturally bright intensity in F₃ in that part of the scene.Experimentally, this phenomenon was found to be relatively rare: testscenes contained a wide range of materials, natural and man-made, yetthis phenomenon only occurred in a few locations. It is not the dominantsource of signal in F₃, as demonstrated by FIG. 3 (bottom). Where thisphenomenon does occur, it can produce some minor purple artifacts. Inaddition, other people observing subjects during the photograph may seea glow from the clothing, thus making the flash slightly visible tothem, although the subjects themselves should not notice this if lookingat the camera.

Exemplary Photometric Flash Measurements

An object of certain exemplary embodiments of dark flash is that itshould be as unnoticeable as possible to human subjects. In certainexemplary experiments, the dark flash output was measured with aspectrometer to determine the spectral irradiance (shown in FIG. 2(b))approximately 1 m from the flash. This was then converted to photometricunits, using the photopic luminosity function as described in, e.g.,Vos, J., Colorimetric and photometric properties of a 2-deg fundamentalobserver, Color Research and Application 125-128 (1978). The luminousexposure for the dark flash was approximately 1.6 lux seconds. A visibleflash set to produce an image V of similar intensity to a dark flashimage F had luminous exposure of approximately 362 lux seconds, a factorof approximately 226 times brighter. This ratio agrees closely with theexperiment results shown in FIG. 3(b), where an attenuation ofapproximately 220 times was used to make the visible flash of comparablebrightness to the dark flash. In FIG. 5, exemplary images D are shownthat were obtained with a visible flash attenuated by this factor. Theresulting images can be unacceptably noisy.

Subjectively, some indications can exist that, when looking directly atthe flash, subjects can see a weak purple light that does not dazzle orleave an afterimage. It was also reported that if not looking directlyat the dark flash, the burst of light was often not seen. By contrast,when using a visible flash that provides a comparable scene exposure,the burst of light can be highly dazzling and leave a strongafter-image.

Exemplary Estimation of Spectral Reflectance

By taking two images, one with the dark flash, the other with a visibleflash, it is possible to obtain 5 different spectral measurements ateach point in the scene, e.g., UV, B, G, R and IR, as opposed to 3obtained with a conventional camera. The spectral reflectances of realworld materials can be accurately modeled in a low-dimensional subspaceusing PCA with relatively few components (see, e.g., Wandell, B. A.,Foundations of Vision, Sinauer Associates (1995)). Using a spectrometerand reflectance probe, it is possible to measure approximately 255different materials in the real world and compute a set of 5 PCA basisfunctions for the range of approximately 360-800 nm. It can then bepossible to use the constrained least squares formulation described in,e.g., Park et al., Multispectral Imaging Using Multiplexed Illumination,ICCV 1-8 (2007), to solve the spectral reflectance functions for thepoints in the scene (S(p, λ) in Equation 4, for example.

FIGS. 10(a) and 10(b) show exemplary graphs generated when using adark/visible flash pair being able to accurately infer the spectralreflectance of objects. For example, FIG. 10(a) shows a spectra of fourdifferent squares 1011, 1012, 1013, 1014, from the example of FIG. 6.The solid lines 1021, 1022, 1023, 1024 show the inferred spectrum. Thedashed lines 1031, 1032, 1033, 1034 show the ground truth. FIG. 10(b)shows RMS estimation errors 1051 for all 24 squares in the color chart650 of FIG. 6 over an approximately 400-700 nm range, compared toexample results 1052 of a multi-spectral illumination approach asdescribed in, e.g., Park et al., supra. As shown, it is possible toaccurately infer the spectrum beyond the visible range. A similar totalerror was achieved with an exemplary dark flash embodiment to theapproach described in Park et al., supra, e.g., approximately 0.82 and0.79 respectively, compared to approximately 1.19 when using R, G, Bchannels alone, for example.

Exemplary Color-Band Denoising

The spectral constraints used in an exemplary dark flash approach can beapplied to images captured by standard cameras. One exemplaryillustration, as shown in FIG. 11, is for conventional flash/no-flashprocessing, using a visible flash/ambient pair.

FIG. 11 shows exemplary images of the model in Equation 5 being used ina visible flash/no-flash setting. For example, FIG. 11 illustrates thetwo exemplary crops 1110, 1120 from group 610 of FIG. 6, with the centerrow 1102 showing the blue channel of the first row 1101. R-Vis 1131illustrates reconstruction with an exemplary model using spectralconstraints. CB-Vis 1132 illustrates an example pipeline as describedin, e.g., Petschnigg et al., supra, based on cross-bilateral filter anddetail enhancement. When using an exemplary algorithm in accordance withthe present disclosure in this configuration, the spectral constraintcan reduce to a single term linking each channel in the flash image toits corresponding channel in the ambient. Thus, the term no longer linksbetween different spectral bands. As shown, the example according to thepresent disclosure yielded better results than the crossbilateral basedmethod example.

Another example of an application is where one color channel is muchnoisier than the others. For example, candle-light is very weak in theblue part of the spectrum compared to red and green. Hence whenattempting to white balance a candle-lit image, the blue channel shouldbe multiplied by a large factor, increasing the noise levels. Usingspectral constraints, the blue channel can be denoised using the red andgreen channels (in place of F₁ and F₃ in exemplary Equation 5). This canprovide a superior result to denoising the blue channel using, e.g.,spatial priors and likelihood alone.

FIG. 12 shows exemplary close-up images of the scene in group 610 ofFIG. 6 illuminated by candlelight, the candlelit image being capturedwith an unmodified Canon 40D. Panel 1201 shows blue channel ofwhite-balanced ambient shot. There is high noise shown in the ambientimage of panel 1201 that can be due to the lack of blue wavelengths incandle-light. Panel 1202 shows example results of denoising of ambientlight using likelihood and spatial priors only. Panel 1203 shows exampleresults of denoising of ambient light using spectral constraints fromthe red and green channels, in addition to the likelihood and spatialpriors. As shown by this example, the spectral constraints technique cansignificantly improve the results.

Exemplary Discussion

As described and demonstrated herein, an exemplary camera and flashsystem in accordance with the present disclosure can take pictures inlow light conditions using a flash that is substantially less noticeableand disruptive than a conventional one. According to certain exemplaryembodiments, exemplary camera and system can primarily use standardhardware combined with novel image processing techniques, as describedherein for example. The spectral constraints can be a powerful way ofcombining the images, yielding good quality results in low lightconditions. In addition, it has been shown that the hardware andsoftware techniques disclosed and described herein can be used in anumber of other applications.

For example, exemplary hardware described herein can be implemented in anumber of ways to, e.g., achieve the same results while taking only oneimage of a scene, allowing for the capture of faster moving scenes andsimplifying the overall complexity of a system in accordance with thepresent disclosure. For example, it can be possible to implementexemplary embodiments in accordance with the present invention using asingle image, by, e.g., modifying the Bayer pattern on the sensor toinclude UV-only and IR-only pixels (e.g., for a total of 5 channels),using certain filters and/or varying the exposure time. Additionally,different flash units can be used to replace the relatively large oneshown herein that have more compact UV and IR LEDs providing a morecontrollable pulse duration and a more precise spectral emission,perhaps further reducing the visibility of the flash. These and otherhardware modifications in contemplation and accordance with the presentdisclosure can also allow for exemplary embodiments of the presentinvention to be implemented in small platforms such as, e.g., cellular(cell) phones, where a flash can often be needed due to poor lowlightperformance on account of the small sensor size, for example.

Exemplary Hardware Details

Exemplary experiments described herein can use a standard Nikon 50 mmf/1.8 lens, which can transmit light down to approximately 350 nm andthus may not the limiting factor in the camera's UV response. A MaxMaxCC3 filter was attached to the lens during the experiments describedherein. The filter was used to block IR light above approximately 850nm, which can otherwise distort the colors of the ambient image (as thenaked sensor's response can extend out to approximately 1100 nm).According to some exemplary embodiments, an IR-block filter can be usedthat, e.g., blocks IR light having wavelengths above approximately 800nm. This filter can tend to not block either visible light or the darkflash. The response functions C_(j)(λ) in FIG. 2(a) can include thefilter and lens, for example. The flash that was used in the experimentsdescribed herein is similar to the Nikon SB-14UV, adapted from astandard SB-14 by removing the UV absorbent coating on the Xenon flashtube. A Hoya U360 filter was attached to the flash to filter out visiblelight. The standard visible flash used in comparisons was equipped witha MaxMax CC1 filter to block its significant IR output.

Exemplary Single Image Blur/Flash Low-Light Photography Procedure

Obtaining an image in low light situations where the shutter can be heldopen for a relatively long duration of time to avoid a noisy outputimage can result in image blur due to camera shake (e.g., motion of thephotographers hands) when the camera is hand-held. As discussed hereinabove, removing this blur can be a very difficult problem known as blinddeconvolution. Heretofore existing procedures and/or algorithms forsolving such blind deconvolution problem (see, e.g., Freeman et al.,U.S. Pat. No. 7,616,826 (Nov. 10, 2010)) are generally not robust andcan not handle relatively large blurs.

Disclosed and described herein, for example, is an exemplary procedurefor overcoming the above-described and other deficiencies in solving theblind deconvolution problem. For example, the following procedure,method and/or algorithm can provide a single image solution forlow-light photography using a conventional camera with a minormodification made to the hardware thereof. According to some exemplaryembodiments of the present disclosure, the obtained/recovered image canbe of a scene and/or subjects illuminated by ambient light instead ofpotentially undesirable flash illumination.

In accordance with exemplary embodiments of the present disclosure, itis possible to use a conventional, unmodified camera having a standardflash. As described herein above, a standard flash can, e.g., be a flashconfigured so that the majority of light emitted has a substantiallyeven distribution over the range of approximately 400 nm toapproximately 700 nm). For example, the conventional, unmodified cameracan be, e.g., a DSLR, compact camera or cell phone camera.

An exemplary procedure in accordance with the present disclosure caninclude, e.g., placing a red filter over the flash, which exemplary redfilter can be selected and/or configured to block wavelengths of belowapproximately 630 nm so that, when the exemplary red filter is placedover the flash, the flash can only substantially emit red light (e.g.light having a wavelength of at least approximately 630 nm). As onehaving ordinary skill in the art should appreciate in view of thepresent disclosure, a green or blue filter also can be used inaccordance with the present disclosure, although using a red filter canprovide for better results.

With the flash powered up and ready to fire (e.g., emit light), a singleimage can be obtained, e.g., using a long shutter duration. The flashcan be configured to fire at the beginning of the exposure. As a result,the image obtained/recorded can have (i) blur due to camera shake (e.g.,resulting from motion of the photographers hands) and, superimposed onit, (ii) a red flash image, which can be relatively sharp (e.g.,non-blurred). The obtained/recorded image can also have little or nosubstantial noise. Accordingly, the obtained and/or captured image canhave three channels (e.g., red, green, blue), with the green and bluechannels being blurred (e.g., by camera shake). Red can be asuperposition of a blurry image with a relatively sharp image (e.g., asa result of the flash as disclosed and described herein).

To recover a blur-free output image of a scene and/or one or moresubjects, an exemplary procedure (and/or algorithm) in accordance withthe present disclosure can include and proceed in, e.g., two stages(sub-procedures), disclosed and described as follows.

1. A first exemplary stage/sub-procedure, for example, can be forestimating the blur, which sub-procedure can be called, e.g., anestimation procedure, and include, e.g.,

(a) estimate the flash image in the red channel F as, e.g.,F=red−(green+blue)/2;

(b) combine the green and blue channels to form a blurry image B, whichcan be expressed as, e.g., B=(green+blue)/2; and

(c) using F and B, it is possible to estimate the blur, which can beencoded as a blur kernel k. For example, according to some exemplaryembodiments, this estimation can be performed by minimizing aleast-squares system where k is the unknown and F and B are known. Onehaving ordinary skill in the art should appreciate in view of thepresent disclosure that there are other procedures, methods and/oralgorithms that can be used to perform this estimation aside fromminimizing a least-squares system. As the blur can apply equally to all3 channels (R, G, B), the blur can be substantially constant over theentire image and handled accordingly by exemplary embodiments of thepresent disclosure as disclosed and described herein, for example. Asone having ordinary skill in the art should appreciate in view of thepresent disclosure, exemplary embodiments of the present disclosure canalso address situations were the blur is not substantially constant overthe entire image.

It is possible that the illumination from the flash (which can bepresent in F, for example) can be different than the illumination fromthe ambient scene (which can be present in B, for example). Suchdifference can be a result of, e.g., one or more shadows that can becaused by a flash that would not be present in the ambient scene.

To address this issue, exemplary embodiments in accordance with thepresent disclosure can, e.g., compute a per-pixel illumination ratiomap: e.g., lambda=(F\oplus k)/B, where (F\oplus k) can represent theflash image F blurred by the blur k. The lambda map can then be used toreweight the blurry image B to correct for the illumination change,which can help cover and/or provide for a substantially accurate valueof k, for example.

The exemplary estimation procedure can operate in a multi-scale fashion.For example, it is possible to start with highly sub-sampled versions ofF and B and estimate a low-resolution version of k and/or lambda. F andB can then be upsampled and used for the initialization of estimationprocedure at the next scale. Performing the exemplary estimationprocedure in this incremental iterative way can be significantly morerobust than an approach attempting to do so directly at the fullresolution, for example.

2. A second exemplary stage/sub-procedure, for example, can be forrecovering the image, which sub-procedure can be called, e.g., arecovering procedure, and include, e.g.:

(a) Performing non-blind deconvolution on green and blue channels usingk to recover sharp versions thereof. To help recover a high qualityresult, the non-blind deconvolution can use, e.g., (i) gradient priors,such as hyper-laplacian gradient priors, and (ii) a constraint that theedges are close to edges of the flash estimate F. Both the exemplarygradient priors and the edge constraint(s) can be efficiently utilizedwith and incorporated into exemplary embodiments of the non-blinddeconvolution procedures disclosed and described herein. According toexemplary embodiments of the present disclosure, this exemplarysub-procedure can be performed in a multi-scale, incremental, iterativemanner, e.g., starting with low-resolution images and incrementallyworking-up to high resolution images through multiple iterations.

(b) In accordance with exemplary embodiments of the present disclosure,the sharp red channel can be recovered by alternating between twosub-procedures. For example, denoting R as the red channelobtained/captured by the camera, it is possible to express R as, e.g.,R=A_r\oplus k+F, where A_r is the desired sharp ambient red image. k canbe estimated from above, and a rough initial estimate of F can be knownfrom, e.g., the expression F=R−(Blue+Green)/2 disclosed and describedabove. Accordingly, the two exemplary sub-procedures that can bealternated between in accordance with exemplary embodiments of thepresent disclosure can be, e.g., disclosed and described as follows:

(i) Estimating A_r by using (R−F) as the blurry image and using theexemplary multi-scale non-blind deconvolution procedure disclosed anddescribed above, for example. According to exemplary embodiments, it ispossible to use, e.g., a hyper-Laplacian prior on the edges of A_r, anda constraint that A_r should be close to A_g and A_b (the exemplarysharp green and green channels recovered above), for example. While theexemplary deconvolution procedure disclosed and described herein can beused, certain other procedures, methods and/or algorithms can be used aswell.

(ii) Refining F by, e.g., enforcing the constraint that F=R−(A_r \oplusk), with the a hyper-laplacian prior on F, in accordance with theexemplary embodiments disclosed and described herein, for example.

As one having ordinary skill in the art should appreciate in view of thepresent disclosure, according to some exemplary embodiments of thepresent disclosure, such as the exemplary single image blur/flashlow-light photography procedure disclosed and described herein, it ispossible that recovering the green and blue channels of the image can berelatively easy in comparison to recovering the red channel since it caninvolve two images and thus exemplary procedures for doing so, asdisclosed and described herein for example, can be more complex.

Exemplary Safety Calculations

With further respect to the safety calculations summarized herein above,the exemplary threshold limit values (TLVs) for UV radiation ofapproximately 180-400 nm incident on the eye (generally considered to bethe most sensitive part of the human body with respect to UV radiation)over an approximately 8 hour period can be provided by the formula on p.155 of TLVs 2001 edition, reproduced as follows in Equation 6:

$\begin{matrix}{{{Max}\mspace{14mu} {flashes}} = \frac{3000}{E_{Eff}}} & (6)\end{matrix}$

Equation 6 can relate the maximum number of flashes to the effectiveirradiance EEff, relative to a monochromatic source at approximately 270nm. EEff can be computed, using Equation 7, from the spectral irradianceof the flash I^(f)(λ) (units: μJ/cm²/nm/flash) and a hazard weightingfunction H(λ) (which is approximately 1 at approximately 270 nm),provided on p. 157 of TLVs 2001 edition.

E _(Eff) =∫H(λ)I ^(f)(λ)dλ  (7)

FIG. 13 shows an exemplary graph of I^(f)(λ) 1301 and H(λ) 1302.Integrating over the product of the two and inserting EEff into Equation6 yields a value of approximately 130,000 flashes. This number scaleswith the inverse square of distance, so at approximately 2 m, the maxsafe limit would be approximately 520,000 flashes, for example.

Exemplary Procedure

Algorithm 1 Fast image deconvolution using hyper-Laplacian priorsRequire: Blurred image y, kernel k, regularization weight λ, exponent α( 

 0) Require: β regime parameters: β₀, β_(Inc), β_(Max) Require: Numberof inner iterations T.  1: β = β₀, x = y  2: Precompute constant termsin Eqn. 4.  3: while β < β_(Max) do  4: iter = 0  5: for i = 1 to T do 6: Given x, solve Eqn. 5 for all pixels using a LUT to give w  7: Givenw, solve Eqn. 4 to give x  8: end for  9: β = β_(Inc) · β 10: end while11: return Deconvolved image x

Following is an example introduction and description of the non-blinddeconvolution problem. In this example, x is the original uncorruptedlinear grayscale image of N pixels, and y is an image degraded by blurand/or noise, which can be produced by convolving x with a blur kernel kand adding zero mean Gaussian noise. For example, y and k can be givenand seek to reconstruct x. Given the ill-posed nature of the task, it ispossible to regularize using, e.g., a penalty function |.|^(α) that canact on the output of a set of filters f₁, . . . , f_(j) applied to x. Aweighting term λ can control the strength of the regularization. From aprobabilistic perspective, it is possible to seek the MAP estimate of x:p(x|y, k) α p(y|x, k)p(x), the first term being a Gaussian likelihoodand second being the hyper-Laplacian image prior. Maximizing p(x|y, k)can be equivalent to minimizing the cost −log p(x|y, k):

$\begin{matrix}{\min\limits_{x}{\sum\limits_{i = 1}^{N}\; \left( {{\frac{\lambda}{2}\left( {{x \oplus k} - y} \right)_{i}^{2}} + {\sum\limits_{j = 1}^{J}\; {\left( {x \oplus f_{j}} \right)_{i}}^{\alpha}}} \right)}} & (8)\end{matrix}$

where i is the pixel index, and ⊕ is the 2-dimensional convolutionoperator. For simplicity, it is possible to use two first-orderderivative filters f₁=[1 −1] and f₂=[1 −1]^(T), although additional onescan easily be added, e.g. learned filters (see, e.g., Osindero et al.,Topographic product models applied to natural scene statistics, NeuralComputation (1995) and Roth et al., supra) or higher order derivatives.For, e.g., brevity, it is possible to denote F_(i) ^(j)x≡(x⊕f_(j))_(i)for j=1, . . . , J.

Using the half-quadratic penalty method (see, e.g., Geman and Reynolds,supra, Geman and Yang, supra, and Wang, Y. et al., supra), it ispossible to introduce auxiliary variables w_(i) ¹ and w_(i) ² (togetherdenoted as w) at each pixel that allow us to move the terms F_(i) ^(j)xoutside the |.|^(α) expression, giving a new cost function:

$\begin{matrix}{\min\limits_{x,w}{\sum\limits_{i}\; \left( {{\frac{\lambda}{2}\left( {{x \oplus k} - y} \right)_{i}^{2}} + {\frac{\beta}{2}\left( {{{{F_{i}^{1}x} - w_{i}^{1}}}_{2}^{2} + {{{F_{i}^{2}x} - w_{i}^{2}}}_{2}^{2}} \right)} + {w_{i}^{1}}^{\alpha} + {w_{i}^{2}}^{\alpha}} \right)}} & (9)\end{matrix}$

where β is a weight that can be varied during the optimization process,as described herein above. As β→∞, the solution of Equation 9 convergesto that of Equation 8. Minimizing Equation 9 for a fixed β can beperformed by alternating between two steps, one where it is possible tosolve for x, given values of w and vice-versa. One exemplary aspect ofthe exemplary procedure according to the present disclosure lies in thew sub-problem, but first briefly described herein is the x sub-problemand its relatively straightforward solution.

Exemplary x Sub-Problem

Given a fixed value of w from the previous iteration, Equation 9 isquadratic in x. Thus, the optimal x can be:

$\begin{matrix}{{\left( {{F^{1^{T}}F^{1}} + {F^{2^{T}}F^{2}} + {\frac{\lambda}{\beta}K^{T}K}} \right)x} = {{F^{1^{T}}w^{1}} + {F^{2^{T}}w^{2}} + {\frac{\lambda}{\beta}K^{T}y}}} & (10)\end{matrix}$

where Kx≡x⊕k.

Assuming circular boundary conditions, it is possible to apply 2D FFT'swhich can diagonalize the convolution matrices F¹, F², K, making itpossible to find the optimal x directly:

$\begin{matrix}{x = {\mathcal{F}^{- 1}\left( \frac{\begin{matrix}{{{\mathcal{F}\left( F^{1} \right)}^{*} \circ {\mathcal{F}\left( w^{1} \right)}} + {{\mathcal{F}\left( F^{2} \right)}^{*} \circ {\mathcal{F}\left( w^{2} \right)}} +} \\{\left( {\lambda/\beta} \right){{\mathcal{F}(K)}^{*} \circ {\mathcal{F}(y)}}}\end{matrix}}{\begin{matrix}{{{\mathcal{F}\left( F^{1} \right)}^{*} \circ {\mathcal{F}\left( F^{1} \right)}} + {{\mathcal{F}\left( F^{2} \right)}^{*} \circ {\mathcal{F}\left( F^{2} \right)}} +} \\{\left( {\lambda/\beta} \right){{\mathcal{F}(K)}^{*} \circ {\mathcal{F}(K)}}}\end{matrix}} \right)}} & (11)\end{matrix}$

where * is the complex conjugate and denotes component-wisemultiplication. The division can also be performed component-wise.Solving Equation 11 uses only 3 FFT's at each iteration since many ofthe terms can be precomputed. The form of this sub-problem can be thesame to that as described in, e.g., Wang, Y. et al., supra.

Exemplary w Sub-Problem

Given a fixed x, finding the optimal w can consist of solving 2Nindependent 1D problems of the form:

$\begin{matrix}{w^{*} = {{\arg {\min\limits_{w}\; {w}^{\alpha}}} + {\frac{\beta}{2}\left( {w - v} \right)^{2}}}} & (12)\end{matrix}$

where v≡F_(i) ^(j)x.

Two exemplary approaches/procedures to finding w* in accordance with thepresent disclosure can be described as follows.

Exemplary Lookup Table (LUT)

For a fixed value of α, w* in Equation 12 can depend only on twovariables, β and v, hence can easily be tabulated off-line to form alookup table. It is possible to numerically solve Equation 12 for 10,000 different values of v over the range encountered in this exemplaryproblem (−0.6≦v≦0.6). This can be repeated for different β values, suchas for integer powers of √2 between 1 and 256. Although the LUT gives anapproximate solution, it can allow the w sub-problem to be solved veryquickly for any α>0.

Exemplary Analytic Solution

For some specific values of a, it is possible to derive exact analyticalsolutions to the sub-problem. For α=2, the sub-problem is quadratic andthus relatively easily solved. If α=1, Equation 12 can reduce to a 1-Dshrinkage operation (see, e.g., Wang, Y. et al., supra). For certainspecial cases of 1<α<2, there can exist analytic solutions (see, e.g.,Wright, et al., Sparse reconstruction by separable approximation, IEEETrans. Signal Processing (2009)). Here, it is possible to address themore challenging case of α<1 and now described is an exemplary procedureto solve Equation 12 for two exemplary cases of α=½ and α=⅔. Fornon-zero w, setting the derivative of Equation 12 w.r.t w to zero canprovide:

α|w| ^(α-1)sign(w)+β(w−v)=0  (13)

For α=½, this becomes, with successive simplification:

|w| ⁻1/2sign(w)+2β(w−v)=0  (14)

|w| ⁻¹=4β²(v−w)²  (15)

w ³−2vw ² +v ² w−sign(w)/4β²=0  (16)

Equation 16 may first appear to be two different cubic equations withthe ±¼β² term. However, it is possible to consider that only one ofthese as v is fixed and that w* lies between 0 and v. Thus, sign(v) canreplace sign(w) in Equation 16:

w ³−2vw ² +v ² w−sign(v)/4β²=0  (17)

For α=⅔, using a similar derivation yields:

$\begin{matrix}{{w^{4} - {3{vw}^{3}} + {3v^{2}w^{2}} - {v^{3}w} + \frac{8}{27\beta^{3}}} = 0} & (18)\end{matrix}$

there being no sign(w) term as it conveniently cancels in this case.Thus, w*, the solution of Equation 12, can be, e.g., either 0 or a rootof the cubic polynomial in Equation 10 for α=½, or equivalently a rootof the quartic polynomial in Equation 17 for α=⅔. Although it can betempting to attempt the same manipulation for α=¾, doing so can resultin a 5th order polynomial, which can thus be solved numerically.

Finding the roots of the cubic and quartic polynomials: Analyticformulae exist for the roots of cubic and quartic polynomials (see,e.g., E. W. Weisstein, Cubic formula, available athttp://mathworld.wolfram.com/CubicFormula.html (last accessed Jan. 7,2010), and E. W. Weisstein, Quartic equation, available athttp://mathworld.wolfram.com/QuarticEquation.html (last accessed Jan. 7,2010)) and they can form the basis of the exemplary approach detailed inexemplary Algorithms/Procedures 2 and 3, for example.

Algorithm 2: Solve Eqn. 5 for α = ½ Require: Target value v, Weight β 1: ∈ = 10⁻⁶  2: {Compute intermediary terms m,t₁,t₂,t₃}  3: m =−sign(v)/4β²  4: t₁ = 2v/3  5: t₂ = ³{square root over (−27m − 2v³ +3{square root over (3)}{square root over (27m² + 4mv³)})}  6: t₃ = v²/t₂ 7: {Compute 3 roots, r₁,r₂,r_(s):}  8: r₁ = t₁ + 1/(3 · 2^(1/3)) · t₂ +2^(1/3)/3 · t₃  9: r₂ = t₁ − (1 − {square root over (3i)})/(6 · 2^(1/3))· t₂    − (1 + {square root over (3i)})/(3 · 2^(2/3)) · t₃ 10: r_(s) =t₁ − (1 + {square root over (3i)})/(6 · 2^(1/3)) · t₂    − (1 − {squareroot over (3i)})/(3 · 2^(2/3)) · t₃ 11; {Pick global minimum from(0,r₁,r₂,r_(s))} 12: r = [r₁,r₂,r₃] 13: c₁ = (abs(imag(r)) < ∈) {Rootmust be real} 14: c₂ = real(r)sign(v) > (⅔ · abs(v)) {Root must obeybound of Eqn. 13} 15: c₃ = real(r)sign(v) < abs(v) {Root < v} 16: w*=max((c₁&c₂&c₃)real(r)sign(v))sign(v) return w*

Algorithm 3: Solve Eqn. 5 for α = ⅔ Require: Target value v, Weight β 1: ∈ = 10⁻⁶  2: {Compute intermediary terms m,t₁,...,t₇:}  3: m =8/(27β³)  4: t₁ = − 9/8 · v²  5: t₂ = v³/4  6: t₃ = −⅛ · mv²  7: t₄ =−t₃/2 + {square root over (−m³/27 + m²v⁴/256)}  8: t₅ = ³{square rootover (t₄)}  9: t₆ = 2(− 5/18 · t₁ + t₅ + m/(3 · t₅)) 10: t₇ = {squareroot over (t₁/3 + t₆)} 11: {Compute 4 roots, r₁,r₂,r₃,r₄:} 12: r₁ =3v/4 + (t₇ + {square root over (−(t₁ + t₆ + t₂/t₇)))}/2 13: r₂ = 3v/4 +(t₇ − {square root over (−(t₁ + t₆ + t₂/t₇)))}/2 14: r₃ = 3v/4 + (−t₇ +{square root over (−(t₁ + t₆ + t₂/t₇)))}/2 15: r₄ = 3v/4 + (−t₇ −{square root over (−(t₁ + t₆ + t₂/t₇)))}/2 16: {Pick global minimum from(0,r₁,r₂,r₃,r₄)} 17: r = [r₁,r₂,r₃,r₄] 18: c₁ = (abs(imag(r)) < ∈) {Rootmust be real} 19: c₂ = real(r)sign(v) > (½ · abs(v)) {Root must obeybound in Eqn. 13} 20: c₃ = real(r)sign(v) < abs(v) {Root < v} 21: w* =max((c₁&c₂&c₃)real(r)sign(v))sign(v) return w*

In both the exemplary cubic and quartic cases, the computationalbottleneck can be the cube root operation. An alternative way of findingthe roots of the polynomials Equation 17 and Equation 18 is to use anumerical root-finder, such as, e.g., Newton-Raphson. In exemplaryexperiments, Newton-Raphson was found to be slower and less accuratethan either the exemplary analytic method or the LUT approach inaccordance with the present disclosure.

Selecting the Correct Roots:

Given the roots of the polynomial, it is possible to determine which onecorresponds to the global minima of Equation 12. For example, when α=½,the resulting cubic equation can have, e.g., (a) 3 imaginary roots, (b)2 imaginary roots and 1 real root, or (c) 3 real roots. In the case of(a), the |w|^(α) term means Equation 12 has positive derivatives around0 and the lack of real roots implies the derivative can not becomenegative, thus w*=0. For the case of (b), the costs of the single realroot and w=0 are compared, an operation that can be efficientlyperformed using, e.g., Equation 20 (below).

In the (c) case, it is possible to have 3 real roots. Examining Equation14 and Equation 15, it can be seen that the squaring operation canintroduce a spurious root above v when v>0, and below v when v<0. Thisroot can be ignored in this example since w* lies between 0 and v. Thecost function in Equation 19 (below) has a local maximum near 0 and alocal minimum between this local maximum and v. Thus, of the 2 remainingroots, the one further from 0 can have a lower cost. The cost of thisroot can be compared with that of w=0 using, e.g., Equation 20.

Similar analysis and reasoning for the α=⅔ case can be used. Forexample, it is possible to potentially have: (a) 4 imaginary roots, (b)2 imaginary and 2 real roots or (c) 4 real roots. In the case of (a),w*=0 is the only solution. For the (b) case, it is possible to selectthe larger of the 2 real roots and compare the costs with w=0 using,e.g., Equation 20, similar to the case of 3 real roots for the cubic. Itis possible that case (c) can never occur in this example with the finalquartic polynomial Equation 18 being derived with a cubing operationfrom the analytic derivative. This can introduce 2 spurious roots intothe final solution, both of which can be imaginary. Thus, it is possiblethat only cases (a) and (b) can be possible in this example.

In both the cubic and quartic cases, an efficient way to pick betweenw=0 and a real root that is between 0 and v can be used. Now describedis a direct mechanism for doing this without involving the relativelyexpensive computation of the cost function in Equation 12, which caninvolve the calculation of a fractional power that can be slow,particularly if, e.g., α=⅔.

For example, r can be the non-zero real root. 0 can be chosen if it haslower cost in Equation 12. This can imply:

$\begin{matrix}{{{{r}^{\alpha} + {\frac{\beta}{2}\left( {r - v} \right)^{2}}} > \frac{\beta \; v^{2}}{2}}{{{{{{sign}(r)}{r}^{\alpha - 1}} + {\frac{\beta}{2}\left( {r - {2v}} \right)}} \lessgtr 0},{r \lessgtr 0}}} & (19)\end{matrix}$

Since it can be possible to only consider roots of the polynomial inthis example, Equation 13 can be used to eliminate sign (r)|r|^(α-1)from Equation 13 and Equation 19, yielding the condition:

$\begin{matrix}{{r \lessgtr {2v\frac{\left( {\alpha - 1} \right)}{\left( {\alpha - 2} \right)}}},{v \gtrless 0}} & (20)\end{matrix}$

since sign(r)=sign(v). So w*=r if r is between 2v/3 and v in the α=½case or between v/2 and v in the α=⅔ case. Otherwise w*=0. Using thisexemplary result, picking w* can be efficiently coded, e.g. lines 12-16of exemplary Algorithm/Procedure 2. Overall, the exemplary analyticapproach/procedure can be slower than the exemplary LUT, but theexemplary analytic approach/procedure can provide a more exact solutionto the w sub-problem, for example.

Summary of Exemplary Procedure

The following in a description of the exemplary embodiment of theprocedure using an exemplary LUT for the w sub-problem. Referring againto FIG. 14, as outlined in exemplary Algorithm/Procedure 1, it ispossible to minimize Equation 9 by alternating the x and w sub-problemsT times before increasing the value of ft and repeating. Starting with asmall value β₀ it is possible to scale it by a factor β_(Inc) until itexceeds some fixed value β_(max). In practice, it can be found that asingle inner iteration can suffice (T=1), although more can sometimes beneeded when β is small.

As with any non-convex optimization problem, it can be difficult toderive any guarantees regarding the convergence of exemplaryAlgorithm/Procedure 1. However, it can be possible to confirm that theglobal optimum of each sub-problem will be found, given the fixed x andw from the previous iteration. Like other methods that can use this formof alternating minimization (see, e.g., Geman and Reynolds, supra, Gemanand Yang, supra, and Wang, Y. et al., supra), there can be littletheoretical guidance for setting the β schedule. It is possible to findthat the simple scheme shown in exemplary Algorithm/Procedure 1 can workwell to minimize Equation 9 and its proxy Equation 8. The experimentsdescribed herein show that exemplary embodiments according to thepresent disclosure can achieve similar signal-to-noise ratio (SNR)levels to iteratively reweighted least squares (IRLS) based approaches,but at a substantially lower computational cost, for example.

Exemplary Experiments

It is possible to evaluate the deconvolution performance of certainexemplary algorithms/procedures on images, comparing them to numerousother methods: e.g., (i) l₂ (Gaussian) prior on image gradients; (ii)Lucy-Richardson (see, e.g., W. Richardson, Bayesian-based iterativemethod of image restoration 62, 55-59 (1972)); (iii) the algorithmdescribed in, e.g., Wang et al., supra, using a total variation (TV)norm prior and (iv) a variant of the algorithm described in, e.g., Wanget al., supra, using an l₁ (Laplacian) prior; and (v) the IRLS approachdescribed in, e.g., Levin, Fergus, Durand and Freeman, supra, using ahyper-Laplacian prior with α=½, ⅔, ⅘. In this example, only IRLS and theexemplary procedure according to the present disclosure use a prior withα<1. For the IRLS scheme, it is possible to use, e.g., theimplementation described in, e.g., Levin, Fergus, Durand and Freeman,supra, with default parameters but with the removal of higher orderderivative filters to enable a direct comparison with other approaches.For example, IRLS and l₂ can directly minimize Equation 8, while certainexemplary embodiments in accordance with the present disclosure, and theTV and l₁ approaches described in, e.g., Wang et al., supra, canminimize the cost in Equation 9, using, e.g., T=1, β₀=1, β_(Inc)=2√2,β_(Max)=256. In this exemplary embodiment, α=½ and α=⅔ can be used, andthe performance of the LUT and analytic methods can be compared as well.For example, the runs can be performed with multithreading enabled(e.g., over 4 CPU cores).

It is possible to evaluate the exemplary procedures using a set ofblurry images, created in the following way, for example. In certainexemplary experiments, 7 in-focus grayscale real-world images weredownloaded from the world wide web. The images were then blurred byreal-world camera shake kernels as described in, e.g., Levin, Weiss,Durand and Freeman, Understanding and evaluating blind deconvolutionalgorithms, CVPR (2009). 1% Gaussian noise was added, followed byquantization to 255 discrete values. In a practical deconvolutionsetting, the blur kernel can not be perfectly known. Therefore, thekernel passed in the exemplary procedures was a minor perturbation ofthe true kernel to mimic kernel estimation errors. In certain exemplaryexperiments with non-perturbed kernels, the results can be similar tothose in exemplary Tables 3 and 1, but with slightly higher SNR levels,for example.

FIG. 14 shows exemplary crops from two images 1401, 1402 beingdeconvolved by 4 different algorithms, including one in accordance anexemplary embodiment of the present disclosure using a 27×27 kernel.Inset 1403 shows the original kernel as described in, e.g., Levin,Weiss, Durand and Freeman, supra, and the perturbed version 1404provided to the algorithms. In this example, the exemplary evaluationmetric was the SNR between the original image {circumflex over (x)} andthe deconvolved output x, which can be defined as, e.g.,

${10\log_{10}\frac{{{\hat{x} - {\mu \left( \hat{x} \right)}}}^{2}}{{{\hat{x} - x}}^{2}}},$

μ({circumflex over (x)}) being the mean of {circumflex over (x)}.

Using exemplary Table 1, it is possible to compare the exemplaryprocedures on 7 different images, all blurred with the same 19×19kernel. For each exemplary procedure, it is possible to search overdifferent regularization weights λ to find the value that gives the bestSNR performance, as reported in the table. In Table 3, it is possible toevaluate the exemplary procedures with the same 512×512 image blurred by8 different kernels (see, e.g., Levin, Weiss, Durand and Freeman, supra)of varying size. Again, the optimal value of λ for each kernel/exemplaryprocedure combination can be chosen from a range of values based on SNRperformance. Exemplary Table 2 shows the running time of severalexemplary procedures on images up to 3072×3072 pixels. FIG. 14 (bottompart) shows a larger 27×27 blur being deconvolved from two exampleimages, comparing the output of different methods.

FIGS. 15(a) and 15(b) show an exemplary image and graph illustratingthat a hyper-Laplacian with exponent α=⅔ can be a better model of imagegradients than a Laplacian or a Gaussian model. For example, FIG. 15(a)shows an exemplary image 1500 of a typical real-world scene. FIG. 15(b)shows an exemplary empirical distribution of gradients 1501 in thescene, along with a Gaussian fit 1502, a Laplacian fit 1503 and ahyper-Laplacian with α=⅔ 1504. As shown, the hyper-Laplacian can fit theempirical distribution closely, particularly in the tails.

The exemplary tables and figures show that an exemplary procedure withα=⅔ and IRLS with α=⅘ can yield higher quality results than othermethods. However, the exemplary procedure can be approximately 70 to 350times faster than IRLS depending on whether the analytic or LUT methodis used, for example. This speedup factor can be independent of imagesize, as shown by Table 2. The l₁ method described in, e.g., Wang, Y. etal., supra, can be considered to be the best of the other methods beingof comparable speed to certain exemplary procedure but achieving lowerSNR scores. The SNR results for the exemplary procedure as shown in thisexample can be almost the same whether LUTs or an exemplary analyticapproach is used. Thus, in practice, the LUT procedure can be preferred,since it can be approximately 5 times faster than the exemplary analyticprocedure and can be used for virtually any value of α.

TABLE 1 IRLS IRLS IRLS Ours Ours Image # Blurry l₂ Lucy TV l₁ α = ½ α =⅔ α = ⅘ α = ½ α = ⅔ 1 6.42 14.13 12.54 15.87 16.18 14.61 15.45 16.0416.05 16.44 2 10.73 17.56 15.15 19.37 19.86 18.43 19.37 20.00 19.7820.26 3 12.45 19.30 16.68 21.83 22.77 21.53 22.62 22.95 23.26 23.27 48.51 16.02 14.27 17.66 18.02 16.34 17.31 17.98 17.70 18.17 5 12.74 16.5913.28 19.34 20.25 19.12 19.99 20.20 21.28 21.00 6 10.85 15.46 12.0017.13 17.59 15.59 16.58 17.04 17.79 17.89 7 11.76 17.40 15.22 18.5818.85 17.08 17.99 18.61 18.58 18.96 Av. SNR gain 6.14 3.67 8.05 8.587.03 7.98 8.48 8.71 8.93 Av. Time (secs) 79.85 1.55 0.66 0.75 354 354354 L: 1.01 L: 1.00 A: 5.27 A: 4.08

Exemplary Table 1 shows an exemplary comparison of SNRs and running timeof 9 different methods for the deconvolution of 7 576×864 images,blurred with the same 19×19 kernel. L=Lookup table, A=Analytic. The bestperforming algorithm for each kernel is shown in bold. In this example,the exemplary embodiment of the procedure according to the presentdisclosure with α=⅔ can beat IRLS with α=⅘, as well as beingsubstantially faster. On average, both of these exemplary methods canoutperform demonstrating the benefits of a sparse prior.

TABLE 2 Image IRLS Ours (LUT) Ours (Analytic) size l₁ α = 4/5 α = 2/3 α= 2/3 256 × 256 0.24 78.14 0.42 0.7 512 × 512 0.47 256.87 0.55 2.28 1024× 1024 2.34 1281.3 2.78 10.87 2048 × 2048 9.34 4935 10.72 44.64 3072 ×3072 22.40 — 24.07 100.42

Exemplary Table 2 shows exemplary run-times of different methods for arange of image sizes, using a 13×13 kernel. The exemplary LUT procedurecan be more than 100 times faster than the IRLS method as described in,e.g., Levin, Fergus, Durand and Freeman, supra.

Exemplary Discussion

Described herein are exemplary image deconvolution schemes that can be,e.g., fast, conceptually simple and yield high quality results.Exemplary procedures can take a novel approach to the non-convexoptimization problem arising from the use of a hyper-Laplacian prior, byusing a splitting approach that can allow the non-convexity to becomeseparable over pixels. Using a LUT to solve this sub-problem allows fororders of magnitude speedup in the solution over existing methods.Additional exemplary embodiments in accordance with the presentdisclosure are available at, e.g., D. Krishnan, Publications,http://cs.nyu.edu/˜dilip/wordpress/ (last accessed Jan. 7, 2010).

Common to the TV and l₁ approaches such as that described in, e.g.,Wang, Y. et al., supra, can use frequency domain operations which canassume circular boundary conditions can be something not present in realimages. These can give rise to boundary artifacts which can be overcometo an extent with edge tapering operations.

Exemplary embodiments in accordance with the present disclosure can beadapted to a range of other problems than those specifically discussedand described herein which, e.g., can rely on natural image statistics.For example, by setting k=1 the exemplary procedure can be used todenoise; or, if k is a defocus kernel, the exemplary procedure can beused for super-resolution. The speed that can be provided by exemplaryprocedures in accordance with the present disclosure can be practical toperform these exemplary operations on the multi-megapixel images frommodern cameras, for example.

TABLE 3 IRLS IRLS IRLS Ours Ours Kernel #/size Blurry l₂ Lucy TV l₁ α =½ α = ⅔ α = ⅘ α = ½ α = ⅔ #1: 13 × 13 10.69 17.22 14.49 19.21 19.4117.20 18.22 18.87 19.36 19.66 #2: 15 × 15 11.28 16.14 13.81 17.94 18.2916.17 17.26 18.02 18.14 18.64 #3: 17 × 17 8.93 14.94 12.16 16.50 16.8615.34 16.36 16.99 16.73 17.25 #4: 19 × 19 10.13 15.27 12.38 16.83 17.2515.97 16.98 17.57 17.29 17.67 #5: 21 × 21 9.26 16.55 13.60 18.72 18.8317.23 18.36 18.88 19.11 19.34 #6: 23 × 23 7.87 15.40 13.32 17.01 17.4215.66 16.73 17.40 17.26 17.77 #7: 27 × 27 6.76 13.81 11.55 15.42 15.6914.59 15.68 16.38 15.92 16.29 #8: 41 × 41 6.00 12.80 11.19 13.53 13.6212.68 13.60 14.25 13.73 13.68 Av. SNR gain 6.40 3.95 8.03 8.31 6.74 7.788.43 8.33 8.67 Av. Time (sec) 57.44 1.22 0.50 0.55 271 271 271 L: 0.81L: 0.78 A: 2.15 A: 2.23

Exemplary Table 3 shows an exemplary comparison of SNRs and run-times of9 different methods for the deconvolution of a 512×512 image blurred by7 different kernels. L=Lookup table, A=Analytic. In this example, theexemplary procedure according to the present disclosure beats all othermethods in terms of quality, with the exception of IRLS on the largestkernel size. However, the exemplary procedure according to the presentdisclosure is faster than IRLS, being comparable in speed to the l₁approach, for example.

FIG. 16 shows a block diagram of a system in accordance with anexemplary embodiment of the present disclosure. For example, anexemplary procedure in accordance with the present disclosure can beperformed by a processing arrangement 1650. Processing arrangement 1650can be, e.g., entirely or a part of, or include, but not limited to, acomputer that includes, e.g., a microprocessor, and use instructionsstored on a computer-accessible medium (e.g., RAM, ROM, hard drive, orother storage device).

As shown in FIG. 16, e.g., a computer-accessible medium 1660 (e.g., asdescribed herein above, a storage device such as a hard disk, floppydisk, memory stick, CD-ROM, RAM, ROM, etc., or a collection thereof) canbe provided (in communication with the processing arrangement 1650). Thecomputer-accessible medium 1660 can contain executable instructions 1670thereon. In addition or alternatively, a storage arrangement 1680 can beprovided separately from the computer-accessible medium 1660, which canprovide the instructions to the processing arrangement 1650 so as toconfigure the processing arrangement to execute certain exemplaryprocedures, processes and methods, as described herein above, forexample.

Further, exemplary processing arrangement 1650 can be provided with orinclude an input arrangement, which can include, e.g., a wired network,a wireless network, the internet, an intranet, a data collection probe,a sensor, etc. Additionally, the exemplary processing arrangement 1650can be provided with or include an output arrangement, which caninclude, e.g., a wired network, a wireless network, the internet, anintranet, etc., as well as a display arrangement and/or a storagearrangement in which data can be stored in a user-accessible formatand/or user-readable format.

FIG. 17 shows a block diagram of another exemplary embodiment of thesystem according to the present disclosure. A computer 1700 can beprovided having a processor 1730 which can be configured or programmedto perform the exemplary steps and/or procedures of the exemplaryembodiments of the techniques described herein above. For example, asubject/specimen 1710 can be positioned and an anatomical region ofinterest can be selected on the subject/specimen 1710. The imagingdevice 1720 can be used to obtain images for the subject/specimen 1710which can be illuminated by an illumination arrangement 1760. Theillumination arrangement 1760 can be controlled by the processor 1730.The data/images can be provided from the imaging device to the computer1700, which can be transmitted to the exemplary processor 1730 and/orstorage arrangement 1740.

According to certain exemplary embodiments of the present disclosure,the data can be stored in a storage arrangement 1740 (e.g., a harddrive, a memory device such as RAM, ROM, memory stick, floppy drive,etc.). The processor 1730 can access the storage arrangement 1740 toexecute a computer program or a set of instructions (stored on or in thestorage arrangement 1740) which can perform the procedures according tothe exemplary embodiments of the present disclosure. Thus, e.g., whenthe processor 1730 performs such instructions and/or computer program,the processor 1730 can be configured to perform the exemplaryembodiments of the procedures according to the present disclosure, asdescribed herein above.

According to other certain illustrative embodiments, a storage medium(or computer-accessible medium containing instructions which receives aplurality of images) can be provided, such as, for example, the storagearrangement 1740. The images can preferably include at least one imagetaken without flash or with ambient light and at least one image that istaken with particular illumination. The particular illumination maypreferably be not visible or partially visible to the subject. At leastone of the illuminated images can be preferably illuminated with atleast one UV frequency or frequency range and at least one of theilluminated images may be illuminated with at least one IR frequency orfrequency range. Instructions can be used by the processor 1730 toprocess the flash and no-flash images according to exemplary proceduresin accordance with the present disclosure as, e.g., described herein.

For example, display 1750 can also be provided for the exemplary systemof FIG. 17. The storage arrangement 1740 and the display 1750 can beprovided within the computer 1700 or external from the computer 1700.The information received by the processor 1730 and the informationdetermined by the processor 1730, as well as the information stored onthe storage arrangement 1740, can be, e.g., stored in the storagearrangement 1740 in a computer-readable/accessible format and/ordisplayed on the display 1750 in a user-readable format.

The foregoing merely illustrates the principles of the presentdisclosure. Various modifications and alterations to the embodimentsdescribed herein will be apparent to those having ordinary skill in theart in view of the teachings herein. It will thus be appreciated thatthose having ordinary skill in the art will be able to devise, e.g.,numerous systems, arrangements, computer-accessible medium andmethods/procedures, which, although not explicitly shown or describedherein, embody the principles of the present disclosure and are thuswithin the spirit and scope of the present disclosure. Further, onehaving ordinary skill in the art will appreciate in view of theteachings provided in the present disclosure, that both the hardware andsoftware elements disclosed herein can be used in a variety of otherapplications aside from those focused on dark flash photography. Inaddition, to the extent that the prior art knowledge has not beenexplicitly incorporated by reference herein above, it is explicitlybeing incorporated herein in its entirety. All publications referencedherein above are incorporated herein by reference in their entireties.

What is claimed is: 1-37. (canceled)
 38. An apparatus for imaging atleast one portion of a target, comprising: at least one light emittingarrangement configured to generate an ultraviolet (UV) light and aninfrared (IR) light to impact the at least one portion; at least onesensor arrangement configured to receive (ii) a further light from theat least one portion that is based on the UV light and the IR lightprovided to the at least one portion and (ii) an ambient light; and acomputer processing arrangement configured to: generate at least onefirst image based on the further light, generate at least one secondimage based on the ambient light, determine a relationship betweenspectral bands corresponding to the at least one first image and the atleast one second image, and generate a third image using the at leastone first image and the at least one second image based on therelationship.
 39. The apparatus according to claim 38, wherein theprocessing arrangement is further configured to identify noiseassociated with the at least one second image based on the relationship.40. The apparatus according to claim 39, wherein the processorarrangement is further configured to at least one of reduce or removethe noise from the at least one second image.
 41. The apparatusaccording to claim 40, wherein the processing arrangement is furtherconfigured to utilize an edge structure of the at least one first imageto remove or reduce the noise.
 42. The apparatus according to claim 38,wherein the processing arranged is further configured to generate thethird image by selecting less than all edges of the at least one secondimage.
 43. The apparatus according to claim 41, wherein the processingarrangement is further configured to select the less than all the edgesin order to avoid at least one at least one shadow or at least onespecularity.
 44. The apparatus according to claim 42, wherein the atleast one shadow is an artifact in the at least one second image. 45.The apparatus according to claim 38, wherein the third image includes aplurality of colors from only the at least one first image.
 46. Theapparatus according to claim 38, wherein a wavelength of the ambientlight is between 400 nanometers and 700 nanometers.
 47. The apparatusaccording to claim 38, wherein a wavelength of the UV light is between360 nanometers and 400 nanometers.
 48. The apparatus according to claim38, wherein a wavelength of the IR light is between 700 nanometers and800 nanometers.
 49. The apparatus according to claim 38, wherein therelationship between spectral bands comprises a correlation betweenspectral bands corresponding to the at least one first image and the atleast one second image.
 50. The apparatus according to claim 38, whereinthe processing arrangement is further configured to generate the thirdimage using a Fast Fourier transform.
 51. The apparatus according toclaim 38, wherein the processing arrangement is further configured togenerate the third image using a lookup-table, and wherein thelookup-table comprises precomputed values stored in a storagearrangement.
 52. The apparatus according to claim 38, wherein theprocessing arrangement is further configured to generate the third imageusing a continuation procedure which repeats until the third image is atleast one of substantially denoised or substantially deconvoluted. 53.The apparatus according to claim 38, wherein the spectral bands include(i) a first spectral band for the IR light, (ii) a second spectral bandfor the UV light, and (iii) a third spectral band for the ambient light,and wherein the first spectral band, the second spectral band and thethird spectral band are different from one another.
 54. An apparatus forimaging at least one portion of a target, comprising: at least one lightemitting arrangement configured to generate an ultraviolet (UV) lightand an infrared (IR) light to impact the at least one portion; at leastone sensor arrangement configured to receive (i) a further light fromthe at least one portion that is based on the UV light and the IR lightprovided to the at least one portion and (ii) an ambient light; and acomputer processing arrangement configured to: determine a relationshipbetween spectral bands corresponding to the further light and theambient light, and generate at least one image using the further lightand the ambient light based on the relationship.
 55. The apparatusaccording to claim 54, wherein the processing arrangement is furtherconfigured to identify noise associated with the further light based onthe relationship.
 56. The apparatus according to claim 55, wherein theprocessor arrangement is further configured to at least one of reduce orremove associated with the further light.
 57. The apparatus according toclaim 54, wherein at least one of (i) a wavelength of the ambient lightis between 400 nanometers and 700 nanometers, (ii) a wavelength of theUV light is between 360 nanometers and 400 nanometers, or (iii) awavelength of the IR light is between 700 nanometers and 800 nanometers.58. A non-transitory computer-accessible medium having stored thereoncomputer executable instructions for imaging at least one portion of atarget, wherein, when the executable instructions are executed by aprocessing arrangement, the processing arrangement is configured toperform procedures comprising: (a) generating an ultraviolet (UV) lightand an infrared (IR) light to impact the at least one portion; (b)receiving (i) a further light from the at least one portion that isbased on the light and the IR light provided to the at least one portionand (ii) an ambient light; (c) determining a relationship betweenspectral bands corresponding to the further light and the ambient light;and (d) generating at least one image using the further light and theambient light based on the relationship.
 59. The computer-accessiblemedium of claim 58, wherein the processing arrangement is furtherconfigured to: generate at least one first image based on the furtherlight; and generate at least one second image based on the ambientlight; wherein the relationship between the spectral bans is determinedbetween spectral bands corresponding to the at least one first image andthe at least one second mage.
 60. A method for imaging at least oneportion of a target, comprising: (a) generating an ultraviolet (UV)light and an infrared (IR) light to impact the at least one portion; (b)receiving (i) a further light from the at least one portion that isbased on the UV light and the IR light provided to the at least oneportion and (ii) an ambient light; (c) determining a relationshipbetween spectral bands corresponding to the further light and theambient light; and (d) using a computer hardware arrangement, generatingat least one image using the further light and the ambient light basedon the relationship.
 61. The method of claim 60, further comprising:generating at least one first image based on the further light; andgenerating at least one second image based on the ambient light; whereinthe relationship between the spectral bans is determined betweenspectral bands corresponding to the at least one first image and the atleast one second mage.
 62. A system for imaging at least one portion ofa target, comprising: a computer hardware arrangement configured to: (a)generate an ultraviolet (UV) light and an infrared (IR) light to impactthe at least one portion; (b) receive (i) a further light from the atleast one portion that is based on the UV light and the IR lightprovided to the at least one portion and (ii) an ambient light; (c)determine a relationship between spectral bands corresponding to thefurther light and the ambient light; and (d) generate at least one imageusing the further light and the ambient light based on the relationship.63. The system of claim 62, wherein the computer hardware arrangement isfurther configured to: generate at least one first image based on thefurther light; and generate at least one second image based on theambient light; wherein the relationship between the spectral bans isdetermined between spectral bands corresponding to the at least onefirst image and the at least one second mage.